Python3 compatible version now available

Python3 compatible version now available - Printable Version

+- SIP (https://nosack.com/sipforum)
+-- Forum: SIP (Sustainable Irrigation Platform) (https://nosack.com/sipforum/forumdisplay.php?fid=1)
+--- Forum: Announcements (https://nosack.com/sipforum/forumdisplay.php?fid=10)
+--- Thread: Python3 compatible version now available (/showthread.php?tid=195)

Pages: 1 2

Python3 compatible version now available - dan - 2019 Dec 01

SIP version 4 is now available.

This is a major upgrade and it is recommended to do a full install rather than upgrade an existing installation. First you may want to rename your existing SIP installation as a backup. This will preserve your settings which can be copied to the new installation. Settings are stored in the SIP/data/ folder and include the files programs.json which holds irrigation schedules, sd.json which holds a wide range of settings and snames.json which holds station names.

From the pi directory you can use the command:

Code:
mv SIP SIP-old

SIP V4 can run under both Python 2.7 and Python 3. The latest version of the Raspbian OS (Buster) still runs Python 2.7 as default as of this writing. Python2 will no longer be supported by the developers after January 1 2020 and Raspbian is expected to switch to Python3 as default in the near future. If you would prefer to run SIP under Python3 there is a file named "sip3.service" in the SIP folder which can be used to enable systemctl to run SIP under Python3

This version of SIP also has a new format for storing irrigation programs. there is a utility program that can convert from the old program format which is stored in the SIP/data/ folder as programs.json to the new format which is stored in the file programData.json.The old programs.json file is left as is as a backup. When SIP V4 is started it tries to load programData.json and if that file is not found it will run the convert_progs.py module which is in the SIP folder. If a programs.json file is not found an empty programData.json file is created ready for new irrigation programs to be defined.

If you have an existing programs.json file you wish to convert to the new format, copy it to the SIP/data folder and do one of the following:

Delete or rename any programData.json file and restart SIP
From the SIP directory run

Code:
sudo python convert_progs.py

form the command line. This will create or overwrite the programData.json file

Many of the most widely used plugins have been updated to work with SIP V4 including:

relay_board
relay_16
The MQTT plugins
weather_level_adj
lcd_adj
email_adj

These plugins will need to be re-installed in SIP V4 and no longer work with SIP V3. Other plugins are being updated as time allows. Please report any problems.

There is also code in place to allow SIP to run under HTTPS but it requires generating a private key and certificate unique to each installation. A key and certificate can generated using openssl. Changing SIP's port setting to 443 on the Options page will activate this feature after the key and certificate are in place. This is an advanced feature and using a self signed certificate will prevent the SIP UI from being accessed with the Chrome browser. You will need to use Firefox. See the code at the end of the sip.py file for locations for the key and certificate files (lines 311 and 312).

Also note that logging is enabled by default and login requiring a passphrase is disabled by default. This is intended to make it easier for new users to get started.

As always, thanks for using SIP and please report any problems.

Dan

RE: Python3 compatible version now available - draggaj - 2019 Dec 03

(sorry I can seem to unbold this font)….Hi Dan,I knocked up a new build SD card and gave it a go...builds as 4.0.2. Found a few relays default high. This is expected till the Relay_board plug in is installed and Hi is set for the number of relays...Installed that plug in, but after installing and enabling (and even rebooting), the plug ins drop down only shows default 3, not Relay_Board, so I cant access the plug in to configure relays to suit as previous. I always add pulse cct as one of my default plug ins. In the drop down I can only see Manage Plugins, Pulse Cct and System Update only. I have not yet tried to add any others such as email or Monthly adjust, yet......rather not complicate this just yet. Any thoughts?Bob Wrote:dan
SIP version 4 is now available.

This is a major upgrade and it is recommended to do a full install rather than upgrade an existing installation. First you may want to rename your existing SIP installation as a backup. This will preserve your settings which can be copied to the new installation. Settings are stored in the SIP/data/ folder and include the files programs.json which holds irrigation schedules, sd.json which holds a wide range of settings and snames.json which holds station names.

From the pi directory you can use the command:

Code:
mv SIP SIP-old

SIP V4 can run under both Python 2.7 and Python 3. The latest version of the Raspbian OS (Buster) still runs Python 2.7 as default as of this writing. Python2 will no longer be supported by the developers after January 1 2020 and Raspbian is expected to switch to Python3 as default in the near future. If you would prefer to run SIP under Python3 there is a file named "sip3.service" in the SIP folder which can be used to enable systemctl to run SIP under Python3

This version of SIP also has a new format for storing irrigation programs. there is a utility program that can convert from the old program format which is stored in the SIP/data/ folder as programs.json to the new format which is stored in the file programData.json.The old programs.json file is left as is as a backup. When SIP V4 is started it tries to load programData.json and if that file is not found it will run the convert_progs.py module which is in the SIP folder. If a programs.json file is not found an empty programData.json file is created ready for new irrigation programs to be defined.

If you have an existing programs.json file you wish to convert to the new format, copy it to the SIP/data folder and do one of the following:
[ul]
[li]Delete or rename any programData.json file and restart SIP[/li]
[li]From the SIP directory run [/li]
[/ul]

Code:
sudo python convert_progs.py

form the command line. This will create or overwrite the programData.json file

Many of the most widely used plugins have been updated to work with SIP V4 including:
[ul]
[li]relay_board[/li]
[li]relay_16[/li]
[li]The MQTT plugins[/li]
[li]weather_level_adj[/li]
[li]lcd_adj[/li]
[li]email_adj[/li]
[/ul]
These plugins will need to be re-installed in SIP V4 and no longer work with SIP V3. Other plugins are being updated as time allows. Please report any problems.

There is also code in place to allow SIP to run under HTTPS but it requires generating a private key and certificate unique to each installation. A key and certificate can generated using openssl. Changing SIP's port setting to 443 on the Options page will activate this feature after the key and certificate are in place. This is an advanced feature and using a self signed certificate will prevent the SIP UI from being accessed with the Chrome browser. You will need to use Firefox. See the code at the end of the sip.py file for locations for the key and certificate files (lines 311 and 312).

Also note that logging is enabled by default and login requiring a passphrase is disabled by default. This is intended to make it easier for new users to get started.

As always, thanks for using SIP and please report any problems.

Dan

RE: Python3 compatible version now available - dan - 2019 Dec 03

Hi Bob,

Did you enable the plugin on the plugin manager page after installing it?

I just installed a fresh copy of SIP on my test Pi and installed the relay_board plugin. After enabling the plugin SIP restarted, the plugin showed up on the drop-down and I was able to access the plugin's set up page.

If you log into the SIP directory and start SIP from the command line with

Code:
python sip.py

you should see a list of the installed plugins.

EDIT: Also the pulse_cct plugin was not up to date with SIP V4. That may be part of the problem.
I updated and tested it. You should re-install it. You can re-install over the old version without uninstalling first.

Dan

RE: Python3 compatible version now available - draggaj - 2019 Dec 04

Hi Dan,
Yep done all that and still same..

disabled and uninstalled relay_board and also pulse, SIP restarted...then reinstalled just relay_board and enabled SIP restarted...same.. see attached GUI snap...

even checked that it was cleaning up in plugins folder...it was..

pi@sprinklers:~/SIP/plugins $ ls -ltrh <--when plug in is disabled and uninstalled...
total 64K
-rwxr-xr-x 1 pi pi 4.5K Dec 3 16:38 system_update.py
-rw-r--r-- 1 pi pi 2.0K Dec 3 16:38 proto.py
-rwxr-xr-x 1 pi pi 6.9K Dec 3 16:38 plugin_manager.py
-rwxr--r-x 1 pi pi 7.0K Dec 3 16:38 mobile_app.py
-rwxr-xr-x 1 pi pi 1.7K Dec 3 16:38 __init__.py
-rw-r--r-- 1 root root 1.6K Dec 3 16:39 __init__.pyc
-rw-r--r-- 1 root root 7.4K Dec 3 16:39 plugin_manager.pyc
-rw-r--r-- 1 root root 4.9K Dec 3 16:39 system_update.pyc
-rw-r--r-- 1 root root 7.8K Dec 3 16:39 mobile_app.pyc
drwxr-xr-x 2 pi pi 4.0K Dec 4 17:19 manifests
pi@sprinklers:~/SIP/plugins $ ls -ltrh <--when plug in is reinstalled and enabled.
total 80K
-rwxr-xr-x 1 pi pi 4.5K Dec 3 16:38 system_update.py
-rw-r--r-- 1 pi pi 2.0K Dec 3 16:38 proto.py
-rwxr-xr-x 1 pi pi 6.9K Dec 3 16:38 plugin_manager.py
-rwxr--r-x 1 pi pi 7.0K Dec 3 16:38 mobile_app.py
-rwxr-xr-x 1 pi pi 1.7K Dec 3 16:38 __init__.py
-rw-r--r-- 1 root root 1.6K Dec 3 16:39 __init__.pyc
-rw-r--r-- 1 root root 7.4K Dec 3 16:39 plugin_manager.pyc
-rw-r--r-- 1 root root 4.9K Dec 3 16:39 system_update.pyc
-rw-r--r-- 1 root root 7.8K Dec 3 16:39 mobile_app.pyc
drwxr-xr-x 2 pi pi 4.0K Dec 4 17:20 manifests
-rw-r-xr-- 1 root root 6.9K Dec 4 17:20 relay_board.py <-----
-rw-r--r-- 1 root root 5.3K Dec 4 17:20 relay_board.pyc <----
pi@sprinklers:~/SIP/plugins $

thanks
bob

(2019 Dec 04, 06:23 AM)If I select Pump control and below in the browse list, they do not appear in the drop down. If I select the Pulse CCt and above, they do appear in the drop down list....I can reinstall the whole shooting match if need be.....let me do another SD card up...draggaj Wrote: Hi Dan,
Yep done all that and still same..

disabled and uninstalled relay_board and also pulse, SIP restarted...then reinstalled just relay_board and enabled SIP restarted...same.. see attached GUI snap...

even checked that it was cleaning up in plugins folder...it was..

pi@sprinklers:~/SIP/plugins $ ls -ltrh <--when plug in is disabled and uninstalled...
total 64K
-rwxr-xr-x 1 pi pi 4.5K Dec 3 16:38 system_update.py
-rw-r--r-- 1 pi pi 2.0K Dec 3 16:38 proto.py
-rwxr-xr-x 1 pi pi 6.9K Dec 3 16:38 plugin_manager.py
-rwxr--r-x 1 pi pi 7.0K Dec 3 16:38 mobile_app.py
-rwxr-xr-x 1 pi pi 1.7K Dec 3 16:38 __init__.py
-rw-r--r-- 1 root root 1.6K Dec 3 16:39 __init__.pyc
-rw-r--r-- 1 root root 7.4K Dec 3 16:39 plugin_manager.pyc
-rw-r--r-- 1 root root 4.9K Dec 3 16:39 system_update.pyc
-rw-r--r-- 1 root root 7.8K Dec 3 16:39 mobile_app.pyc
drwxr-xr-x 2 pi pi 4.0K Dec 4 17:19 manifests
pi@sprinklers:~/SIP/plugins $ ls -ltrh <--when plug in is reinstalled and enabled.
total 80K
-rwxr-xr-x 1 pi pi 4.5K Dec 3 16:38 system_update.py
-rw-r--r-- 1 pi pi 2.0K Dec 3 16:38 proto.py
-rwxr-xr-x 1 pi pi 6.9K Dec 3 16:38 plugin_manager.py
-rwxr--r-x 1 pi pi 7.0K Dec 3 16:38 mobile_app.py
-rwxr-xr-x 1 pi pi 1.7K Dec 3 16:38 __init__.py
-rw-r--r-- 1 root root 1.6K Dec 3 16:39 __init__.pyc
-rw-r--r-- 1 root root 7.4K Dec 3 16:39 plugin_manager.pyc
-rw-r--r-- 1 root root 4.9K Dec 3 16:39 system_update.pyc
-rw-r--r-- 1 root root 7.8K Dec 3 16:39 mobile_app.pyc
drwxr-xr-x 2 pi pi 4.0K Dec 4 17:20 manifests
-rw-r-xr-- 1 root root 6.9K Dec 4 17:20 relay_board.py <-----
-rw-r--r-- 1 root root 5.3K Dec 4 17:20 relay_board.pyc <----
pi@sprinklers:~/SIP/plugins $

thanks
bob

go figure, fresh install using 2018-11-13-raspbian-stretch as my base, identical result......might get a fresher raspian build and try again.

RE: Python3 compatible version now available - draggaj - 2019 Dec 04

nope, same with latest 2019-09-26-raspbian-buster.
burn replacement SD, add SSH file, add wpa_supplicant.conf,install in RPI and boot headless.
find on network and SSH to it, sudo apt-get update, sudo raspi-config, localise, expand filesystem, change hostname and password, reboot. Install GIT and SIP as per instructions...run SIP.....same issue with plugins. Previous finding on browsed plugins is incorrect. Random selection of plugins dont appear in dropdown....i give...

RE: Python3 compatible version now available - dan - 2019 Dec 04

I see the same problem on a different system.

I found a problem with the relay_board plugin. It is fixed now.
A re-install of the plugin should now appear in the drop-down.

RE: Python3 compatible version now available - draggaj - 2019 Dec 05

Excellent....yep that fixed it..many thanks for the assistance.

RE: Python3 compatible version now available - astrogerard - 2019 Dec 07

Dan, first of all, many thanks for all the work you are doing on SIP to get it working under Python3.

Since winter has arrived and the valves are disconnected it is time to migrate to new versions of the OS, Python and SIP. Just to give some feedback about the upgrade proces I must say it went almost without any problem.

On the Raspberry Pi 3 (Model B Rev 1.2) I started with a new raspbian 10 (buster) lite image and ran a headless setup. Then installed Python 3.7.3, python3-RPi.GPI, python3-paho-mqtt, git and of course cloned SIP.

When starting SIP it failed with:

Code:
Traceback (most recent call last):

  File "sip.py", line 6, in <module>

    from future import standard_library

  File "/home/sip/SIP/future/standard_library/__init__.py", line 64, in <module>

    import logging

  File "/usr/lib/python3.7/logging/__init__.py", line 26, in <module>

    import sys, os, time, io, traceback, warnings, weakref, collections.abc

  File "/usr/lib/python3.7/traceback.py", line 3, in <module>

    import collections

  File "/usr/lib/python3.7/collections/__init__.py", line 27, in <module>

    from reprlib import recursive_repr as _recursive_repr

  File "/home/sip/SIP/reprlib/__init__.py", line 7, in <module>

    raise ImportError('\n To run SIP under Python 3 '

ImportError:

To run SIP under Python 3  Please rename the folder reprlib.

In the SIP directory, use the command:

mv reprlib reprlib-bak

After removing reprlib SIP started succesfully.
Then copy/paste'd some config files and ran convert_progs. All went fine
The plugins I use at the moment are: Relay board, MQTT, MQTT Zone Broadcaster and CLI Control.

I noticed after starting manually a station for a certain amount of time the home screen isn't updated. I can't remember if it did in the old version.

So the migration went very well. Thanks again Dan for the good work!

--Gerard

RE: Python3 compatible version now available - dan - 2019 Dec 08

Hi Gerard,

As always it's great to hear from you.

Glad the update went well. There are still a few plugins that need to be updated and tested.

With manual operation the screen does not update. This has been the case from the start. It is on my todo list.

In the future there will be a version of SIP that will be Python 3 only. That will be cleaner code and will not be such a major change as SIP V4.

This was made possible by a new group of maintainers of the Web.py framework that SIP is built on. Web.py development had been stagnant for several years and I found it very exciting to see it moving forward again. Without Web.py being updated to Python 2 + 3 compatibility SIP would still be Python 2 only.

Dan

RE: Python3 compatible version now available - PolarBear56 - 2020 Apr 19

Hi Dan,

Love the program!

Just updated to v4 and followed (I think) all of the steps. I'm having a problem with the convert_progs step. I keep getting a syntax error every time I try to run it and it just creates a blank ProgramData.json file. Is there something I'm missing or haven't done?

Everything else works great.

Cheers!