Copying assets

Why would you want to copy an asset? Well, we learned from building actual projects that a lot of configuration goes into a well-working asset or even a complete site.

You might want to do the same thing over and over again after the asset you configured in a pilot performs really well.

Copy it to a new customer account! Including child assets, sensors, attributes, flex-config. Only the time series data ("beliefs") is not copied. 

In this release, we added API support (/api/v3_0/assets/ID/copy) and a button on the asset page (which creates the copy as a sibling of the original or in your own account).

Follow-up in the next release: Better support for the creation of new assets from (public) templates. Plus, adding some of these templates as in-built starter packs. This should help with the actual goal of creating new sites with known components, even for new users.

This work was done in Pull Requests: 2017 and 2121.

Account sources

Time series data ("beliefs") in FlexMeasures are linked to "data sources". That is a boring topic for many, but it has a strong use case: fetching only data created by the source you are interested in - like a third-party price API, the latest forecasting model etc.

We learned from practice that we needed to organize data sources in a more practical manner. This discussion tells the story of what Felix and Nicolas came up with in that Pizza place one night in Brussels after FOSDEM. We suggested several stories, and one of those stories is now implemented: "Data sources linked to account".

If data sources have a link to an account, then it becomes possible to collect all data which was created by users or scripts belonging to that account.

Here is a scenario: The wholesale price data for the next day comes via a script which uses the ENTSO-E API. Then, reporters compute the retail prices of various suppliers. If that ENTSO-E service is unreliable some day, a user of the FlexMeasures host might log in and upload some prices manually, so that scheduling can continue (the sensor page's upload functionality is rather new and rather handy!). The data source will then be the user who uploaded. In the reporters, we can accept that any user from the host is also an applicable source. The result: A more robust and serviceable system.

For the PRs involved, check the story. Also in PR 2185, we improved how data from different sources are distinguished in graphs.

Data ingestion queue

When data is sent/uploaded to FlexMeasures, there is some business logic involved. FlexMeasures resamples data, for instance, and checks for duplicates. We noticed that this can occupy the API. So all such logic execution is now sent to the workers queue. In fact, next to the scheduling and forecasting queues, FlexMeasures now can fill an ingestion queue. This means you might want to update your worker setup. But if there is no "ingestion" queue, things should still work as normal.

This work was done in Pull Request 2101.

More fine-grained monitoring

Monitoring (of failed CLI tasks or absent users) was in need of more fine-grained controls. Here are the two improvements:

1. You can now control per monitoring filter which users or email addresses get monitoring alerts. Until now, there was just one config setting with a list of recipients.

2. Absent users can now be filtered by account, or by who the consultant is. That is more practical than what we had so far (filter by user or account role).

This work was done in Pull Request 2158.