mqtt plugin question

[astrogerard]

Hi All,

Since this is my first post I'll would like start with a thank you to all of you for creating SIP. I almost started to make my own irrigation system but found SIP just in time.

As a fan of mqtt I want to make intensive use of it. However when the broker is (temporarily) not available, due to system startup or other causes, and returns back online the zone publishing is stopped. Only after restarting SIP the publishing of messages will work again.

Does anyone recognise this issue and even better, knows how to solve this?

Thanks,
--Gerard

------------------
Hardware: Raspberry Pi 3 B
OS: DietPi (Raspbian 8 jessie)
SW: msquitto, node-red, SIP (3.2.5)

[dan]

Assuming you are running both mosquitto and SIP on the same Pi and using system.d unit files to start both programs you should be able to edit the unit file for sip (/etc/systemd/system/sip.system) and set mosquitto as a dependency.

e.g. add a line like:
After=mosquitto.target

near the top of the sip.service unit file.

The idea is to set the boot order so that mosquitto starts before SIP does during a restart of the system. I think that may eliminate the problem.

There is an answer to a similar question here:
https://serverfault.com/questions/482730...boot-order

This link may be even more helpful:
https://fedoramagazine.org/systemd-unit-...and-order/


I have not tested this but I think it could work. If you try it, let us know your result.

Dan

[astrogerard]

Thanks Dan for taking the time to look into this.

For now the broker is locally installed to keep everything together and make things as simple as possible. In the future I want to use the broker I already use for my home automation system (and more) and want to integrate some SIP functions (start/stop/monitor) from there as well.

For the startup problem I tried the After= parameter but that unfortunately did not work well. I don't know why but it seems that when the mosquitto process is up it takes a little more time to become really alive. For now I added ExecStartPre=/bin/sleep 10 within [Service] which works ok but almost doubles the startup time of the Pi. But what's 20 seconds :-)

Hopefully SIP and the plugins will be actively maintained and if so, in time I might contribute some integration functionality.

--Gerard

[dan]

SIP has been under development for over 5 years now and the plan is to keep it updated for the foreseeable future.

When you change to using your other MQTT broker the only thing that might be a problem is if the broker goes down SIP will not be able to re-subscribe without a restart. There may be a way to have sip to check the connection and resubscribe if necessary but that will need some work.

Dan

[astrogerard]

Hi Dan and others,

In these "winter and wet days" I want to try to get the mqtt code more resilient against broker unavailability and hopefully later create some more mqtt extensions to "talk" to the MySensors network. 

Maybe (probably) a stupid question.... Where goes the output of print statements to?  I see some print statements in the mqtt code but even when SIP is started from the command line it doesn't show up. Debugging is a bit difficult without it.

Probably the stupidest question is easy to answer. :-)

Thanks,
--Gerard

[dan]

Hi Gerard,

I just took a quick look at the MQTT plugin code.
The print statements I saw are either in a try - except block or inside a function.

You would need to cause an error (exception) to happen or cause a particular function to be run to see the output.

Try putting your own print statements in code that should run under normal conditions. You should see the output.

You are right that SIP needs to be started from the command line to see the output from print statements.

With plugins that have print statements the print output may show up at the very start of when the plugin is loaded. Check the stuff that appears before the list of plugins loaded.

Dan

[astrogerard]

Thanks Dan,

Indeed print statements are working. Silly me was assuming some code was executed but it wasn't. I was trying to fire a runonce via mqtt (SIP/schedule topic) noticed that the subscription part of the code is not executed and therefore on_message also remains silent.

I will have a look and see if I can get it working.

--Gerard

**EDIT** Found it. Although the topic displayed correctly on the settings page you still have to save the settings first.

[dmonty]

In these "winter and wet days" I want to try to get the mqtt code more resilient against broker unavailability and hopefully later create some more mqtt extensions to "talk" to the MySensors network. 

I'm very interested in extensions that will work with MySensors.  Namely MySensors have some cheap wireless moisture sensors that run on battery+solar.   This would make SIP a very precise irrigation solution being aware of soil-moisture content.   Also My sensors can track wind speed - i.e. don't water when it's too windy as it is not as effective.

In places where water is very scarce/expensive, drip irrigation is coupled with moisture sensors in the ground and nailed into the trunk of trees.

[astrogerard]

Well, thats almost the same I want to do.
In our region there is plenty of water but the requirement on one part of the garden can be totally different from another part of the garden. This all is due to the weather of the past days, the time of year (sun high in summer or low in spring/autum), wind and so on.

Ideally I want moisture measurement for every station and combined with other weather data make a descision if and how long the station must be turned on. Additionally some level of autolearning should also be possible. So the way SIP works now it isn't suitable to do this. Don't get me wrong, I'm still happy with SIP the way it is now.  :-)

At the moment I'm optimising the MQTT plugins so that connectivity/availability of the broker is more failsafe. Within my test environment this is working fine now. I also implemented a hearbeat signal which can trigger every plugin once a minute so the plugin can do some houskeeping if nessecary.

The next step is to enhance the MQTT plugin so that additional (sensor)data or commands can be received. This is quite easy to incorporate but the biggest question is "what to do with this data?"  There must be a kind of logic with some configurable rule engine so that the enduser of SIP can create the required if-then-else logic. I already experimented with some stuff like Node-red, Blockly and Domoticz that I already have running for other purposes. 

Maybe we should start a feature request conversation to first get the functional requirements clear and after that think of a possible solution. Any thoughts on this are welcome of course.

To be continued...

--Gerard

[dan]

An interesting thread!

I think one change that needs to happen in SIP is to modify the structure of the irrigation program data that SIP uses.

Right now it is based on the original OpenSprinkler Arduino code that SIP was originally ported from and was optimized to save space in the Arduino's limited memory.

I had started to convert the programs data structure to a Python dictionary so that each setting could be accessed by a key (eg "duration = 60" to set the duration to 60sec.) this would make it easy to change irrigation parameters on the fly.
That project got sidetracked a while back but I think I will get back on it. That would also make it possible to add new parameters from a plugin.

During the development of SIP I have avoided using a database to minimize writing to the SD card in order to prolong its life. Lately I have been looking at redis an in-memory database that has Python and Nodejs clients (among others) available to interface with it. It might be useful.
https://redis.io/

Version 4.0 and + specifically supports Raspberry Pi.

Dan