v1.1.0: Automations
Nicolas Höning
Version v1.1.0 of FlexMeasures has 177 PRs and one overarching addition: automations. With automations, we upgrade FlexMeasures to become a data pipeline platform.
See the changelog for a complete list of improvements. Here, we cannot talk about everything that happened. We focus on automations. Honorable mentions:
- Keeping better track of data sources
- Rate limits: more usable (after first real-life tests)
- add annotations on sensor data in the UI
- 3x speed increase in test suite
- more usable status page (focus on jobs first)
- various bug, perfomance and UX fixes
Automations
For recurring tasks, e.g. refresh PV generation forecasts in the morning and around noon, or schedule a site (at least) every 4 hours, we used cron jobs. Another alternative was that an outside system (e.g. a local EMS) would take this recurrence over and call the FlexMeasures API regularly. However, that does not work for every use case.
Enter: automations. Using FlexMeasures queuing system as a backbone, users can now automate recurrent runs of forecasts, schedules and reports via the UI:
You can edit re-occurrence patterns and configuration. You can also see whether an automation is active (green light), short statistics of recent jobs and the next run time. You can even run an automation right away.
This is great as management of sites moves out of cron jobs - less command line hacking and log file reading. And: more users can manage how a site runs daily (not just the admins who are allowed on the cron server). This includes not just hosts. Consultants can manage their client's sites, or users manage their own. They can inspect automations together and delegate the management and monitoring of it.
And a cron server is now not a CPU bottleneck. If a host needs to scale, they concentrate on their worker CPUs.
There is a lot of functionality we already worked out under the hood, for instance that FlexMeasures plugins can also add automations for their custom commands - an example would be retrieving data from a 3rd party API. In fact, flexmeasures-weather and flexmeasures-entsoe will soon allow you to automate weather forecast and price retrieval this way!
And, as usual, automations can be managed fully via the FlexMeasures API (and to a big extent in the CLI).
What will we do in v1.2?
- Monitoring - failures need to become alerts (via email), to the right people, in the right amount. Work has started.
- Editable automations - You can only edit the recurrence pattern as of now, but soon also the configuration itself.
- Configuration via forms - now many configuration and parameter details need to be entered as JSON. As we use schemas to validate those in all data generators, we can start to offer UI forms. For reporting templates, we already are working on this.
- Dependencies - some jobs are dependent on others. The queuing supports this, but our automation layer still needs that. For instance: retrieve price data, and if that worked, schedule. This also already has a PR, see screenshot below for proposed UI.
