API Request Scheduler

🔖 scheduler api raspberry-pi golang redfish 

For a technically minded person, having a server at home doesn’t sound too strange. They can be extremely useful for anything from file shares, media servers, to development boxes, etc etc. A friend of mine uses theirs as a backup repository, as well as share files between machines with OwnCloud. Another friend runs VIRL, a virtual routing environment, so that he can build network designs and study networks. However, servers are not designed for home use, therefore pose some challenges when used at home.

Sneak Peek: Overview Of Solution Process Flow

Sneak Peek: Overview of Solution Process Flow

Sneak Peek: Overview Of Solution Process Flow

Sneak Peek: Overview of Solution Process Flow

Problem Overview

Solution Approaches

I have tackled the noise, by sticking it in a cupboard away from my living space, but this is closer to my bedroom, so can be noisy at night. Also, I’m not using it at night, so the noise and power use is not wanted, nor necessary. I saw 3 approaches to manage this:

  1. Manually power ON and OFF the server every day
    • I tried this first, but it wasn’t very effective
    • When needing the server, I realise I’ve forgotten to power it on, then have to wait whilst it boots
    • Booting can take a while with a server, as it checks all components and isn’t designed for quick boots (typically a server never gets shut down, except for maintenance)
    • At night, or leaving the house, I would forget to turn it off
    • No-one wants to get back up once cosy in bed
  2. Using a ‘smart plug’ with a timer or schedule to kill and provide the power when desired
    • Typically they do not have a lot of scheduling flexibility
    • Clunky user interfaces, or none at all (revering to programming the plug itself)
    • This is not a graceful shutdown so it increases risk of data corruption and integrity errors, OS failing to boot, and hardware failures
  3. Use a tool to monitor a schedule, then trigger power ON and OFF commands to the server management API
    • Schedule can be done in a few ways
    • Triggering the commands via API allows me to do graceful shutdowns and protect the server
    • Tool requires access to the API, sitting on the servers ‘powered off management console’ (CIMC, iLO, DRAC)

Option 3 is the direction I took, but I couldn’t see a simple way to do this with a tool that existed. Any service on the Internet would require access to the ‘powered off management console’ of the server, so I would have to setup port forwarding on my edge router (to traverse stateful NAT PAT), setup dynamic DNS with an agent running inside the house (to keep the home IP known externally if the ISP changes this), take more care with security of the servers ‘powered off management console’ and what it speaks back to (having had something similar setup in the past, the logs showed hundreds of automated login attempts on the admin interface, admin:password123, admin:pa$$w0rd123, …, …, …). Lastly, the API uses a self-signed certificate, many ‘more official’ tools expect properly configured certificates, as is best practice, but I do not want to have to manage a certificate on the server API.

So I took a different approach and built something to run within my home, within my LAN, from a RaspberryPi. As this is local to the ‘powered off management console’ I don’t have to set up and manage any of those things mentioned above. For scheduling I can hook into an ICAL, Google Calendar etc, which the RaspberryPi can request from within my home (forming the start of a stateful connection, so no port forwarding or dynamic DNS).

Not going to lie, I also wanted to try out GoLang, and build something cool.

Solution

Components

These components allowed me to get going quickly, work around certain restrictions, and result in something that meets my requirements. Some of this has been detailed above in “Solution Approaches”.

Google Calendar

Everyone knows what Google Calendar is (I hope). It is important to be realistic and efficient, I was not going to make a full calendaring function and host this for scheduling, particularly not of a high quality in the desired short time frame of the project. My initial thoughts were to code in time frames (06:00 to 22:00), but this wasn’t flexible, and would still require myself to create a front end for both PC and mobile.

This is why I chose to use someone else’s calendar, pulling the data via ICS, and making a decision based upon that. Allowing myself to now schedule the status via my PC (Google Calendar WebUI), via my phone (Google Calendar App), control sharing permissions with my family, from anywhere with an Internet connection.

Google Calendar on Android mobile and Web

Google Calendar on Android mobile and Web

Scheduling Event Every Weekday, 1800 to 2200

Scheduling Event Every Weekday, 1800 to 2200

RaspberryPi

This Pi has been sat in a box for many years now, covering different roles, each which have been superseded by changes in needs or updates in tool-sets. For the project the Pi is perfect, running silently hidden away, consuming very little power, capable of running any code I give it. I created my tool with GoLang, which is supported across platforms. GoLang can compile into binary code for x86, ARM, Windows, Linux, BSD, etc. Getting the Pi up and running is a breeze, thanks to the work the RaspberryPi team have done, flashing on the Raspbian OS, enabling SSH and going from there.

RaspberryPi Used in Project

RaspberryPi Used in Project

API Request Scheduler

I created this code specifically for this project, but designed it to be flexible. With the use of a configuration file, it can be pointed at any API (which supports basic authentication), at any calendar which is ICS, and map search strings to API payloads that work with that API. At present it supports up to 5 different API payloads, in future it can be updated to support effectively an unlimited number, but since I only need 2 I am happy with it. My code is open on GitHub, so feel free to look, branch, etc. To have this run regularly, it’s set as a cron job on the Pi.

Please note: I'm not a developer by trade, nor a server/Linux admin, [my background](/about/philip-whiteside) is Networking, so my code etc probably has many strange aspects to it.

ICS-GoLang

A GoLang package imported into my tool, without this it would have taken a lot longer to develop the tool, with a lot more Googling and hair pulling. This is the power of open source! The code is open on GitHub, this is a great piece of work that worked flawlessly for me, thank you to PuloV for this.

RedFish API

Every API is different in someway, thankfully my server supports the RedFish API, which is a standard API for servers day to day operations. This is what allows me to do a graceful shutdown, not a ‘pull the power’, as well as booting up, getting information, etc etc. Nearly all the major server manufacturers support it in some shape or form, but hardware and software versions will need to be checked.

Code

The first part of the code is the simplest, to get the file path of the config file which the user wants to use. By specifying the path users can use any path they may like, rather than staticly mapped, which caused a problem for me (discussed in the Raspbian & Cron section).

Note: Code below has been truncated to show just the area I'm discussing. Also verbose StdOut prints have been removed to keep it clean. For full code please refer to [GitHub](https://github.com/Philiphlop/apiRequestScheduler) 
func main() {
    // READ command-line arguments to get config file path
    // The [1:] skips the first word, the name of the binary
    CliArguments := os.Args[1:]

    // We can now use these arguments like so.
    // This will print it to StdOut
    fmt.Println(CliArguments[0])
}

Now that we know where the config file is, the next step is to load that and be prepared for the users environment. I could have passed all of these in as CLI arguments too, but I generally prefer config files as they are saved, easily editable, easily read, etc.

The first aspect is to create a struct, which will hold the data we pull in from the config file, which can then later be passed to the functions for use. Here I am quite static with the use of 5 string values to payloads; if you were to use much more than 5, you probably wouldn’t want to do it this way and instead be smarter in building this out, otherwise it can be very long.

// Config for user configuration used throughout
type Config struct {
    ServerAPIPath            string
    ServerAPIUser            string
    ServerAPIPass            string
    ServerAPIPayloadDefault  string
    CalendarValue001         string
    ServerAPIPayload001      string
    CalendarValue002         string
    ServerAPIPayload002      string
    CalendarValue003         string
    ServerAPIPayload003      string
    CalendarValue004         string
    ServerAPIPayload004      string
    CalendarValue005         string
    ServerAPIPayload005      string
    ServerAPIIgnoreCertError bool
    ICALPath                 string
}

These map directly to the config file, which is a JSON based file. As you can see the names match between the two, it is possible to map to different names, but I had no need for that. Below is the sampleConfig.json included with the code. We can also see the mappings here, so when the tool see’s an active calendar event with the subject/summary of ON (CalendarValue001), it will trigger the payload of {"ResetType":"On"} (ServerAPIPayload001).

{
    "ServerAPIPath": "https://serverAPI/redfish/v1/Systems/SNabcabcabc/Actions/System.Reset",
    "ServerAPIUser": "admin",
    "ServerAPIPass": "Password123",
    "ServerAPIPayloadDefault": "{\"ResetType\":\"GracefulShutdown\"}",
    "CalendarValue001": "ON",
    "ServerAPIPayload001": "{\"ResetType\":\"On\"}",
    "CalendarValue002": "",
    "ServerAPIPayload002": "",
    "CalendarValue003": "",
    "ServerAPIPayload003": "",
    "CalendarValue004": "",
    "ServerAPIPayload004": "",
    "CalendarValue005": "",
    "ServerAPIPayload005": "",
    "ServerAPIIgnoreCertError": false,
    "ICALPath": "https://calendar.google.com/calendar/ical/xyzxyzxyz/basic.ics"
}

To load this in we add a LoadConfiguration function call into main() and the function itself.

func main() {
    // READ command-line arguments to get config file path
    // The [1:] skips the first word, the name of the binary
    CliArguments := os.Args[1:]

    // Pass first CLI argument as config file location
    // LoadConfiguration then returns the data into the Config struct
    config := LoadConfiguration(CliArguments[0])
}
// LoadConfiguration file then return Config struct
func LoadConfiguration(filename string) Config {
    var config Config
    // Load the configuration file using GoLang's built in os.Open()
    // If there is an error, save this in err, to be displayed later
    configFile, err := os.Open(filename)

    // Wait until the above is completed
    defer configFile.Close()

    // If there is an error, display it on StdOut
    if err != nil {
        fmt.Println(err.Error())
    }

    // Decode the JSON into a format the tool can understand
    jsonParser := json.NewDecoder(configFile)
    jsonParser.Decode(&config)

    // Return the values into the Config struct
    return config
}

Now that we’ve loaded the configuration, we can start doing the fun stuff. The next phase is to pull down the ICAL data to inspect. For this I used PuloV/ics-golang. He provides some good examples in his repository, which is what I used to get started. Again we need to call this function from main().

func main() {
    // READ command-line arguments to get config file path
    // The [1:] skips the first word, the name of the binary
    CliArguments := os.Args[1:]

    // Pass first CLI argument as config file location
    // LoadConfiguration then returns the data into the Config struct
    config := LoadConfiguration(CliArguments[0])

    // LoadICAL and return current event summary ON/OFF
    // This will return an integer number, which I correlate with the config for the appropriate request payload
    // This part is effectively limitless in the number of options you can have
    // We are passing in the config struct, so that the function can see that data
    desiredStatus := LoadICAL(config)
}
// LoadICAL and return current event summary ON/OFF
func LoadICAL(config Config) int {
    // This first part is mostly a copy from the examples provided
    // They are clearly laid out so I won't repeat here
    //  create new parser
    parser := ics.New()
    // set the filepath for the ics files
    ics.FilePath = "tmp/new/"
    // we dont want to delete the temp files
    ics.DeleteTempFiles = false
    ics.RepeatRuleApply = true
    // get the input chan
    inputChan := parser.GetInputChan()
    // send the calendar urls to be parsed
    inputChan <- config.ICALPath
    //  wait for the calendar to be parsed
    parser.Wait()
    // get all calendars in this parser
    cal, _ := parser.GetCalendars()

    // Check ICAL payload for errors
    // I was getting some panic errors, where the data wasn't being pulled correctly
    // I added a check to see if there was data and report accordingly
    if len(cal) > 0 {
        fmt.Println("LoadICAL() - Calender entries have been pulled. Slice is populated")
    } else {
        fmt.Println("LoadICAL() - Calender entries have not been pulled. Slice is empty")
        fmt.Println("LoadICAL() - App will panic")
        fmt.Println("LoadICAL() - Please check the URL of the calendar ICS")
    }

    // Check ICAL current event payload
    // Once we have the payload we can look for the first active event
    // I use a dedicated calendar for this, so there is no concerns of overlapping events etc
    // Again this is not a scalable element, if doing a lot more than 5 options it will start to get very big
    for _, e := range cal[0].GetEvents() {
        now := time.Now()
        if now.After(e.GetStart()) && now.Before(e.GetEnd()) {
            if strings.EqualFold(config.CalendarValue001, e.GetSummary()) {
                fmt.Println("LoadICAL() - Current even is", e.GetSummary(), "-", &e)
                return 1
            } else if strings.EqualFold(config.CalendarValue002, e.GetSummary()) {
                fmt.Println("LoadICAL() - Current even is", e.GetSummary(), "-", &e)
                return 2
            } else if strings.EqualFold(config.CalendarValue003, e.GetSummary()) {
                fmt.Println("LoadICAL() - Current even is", e.GetSummary(), "-", &e)
                return 3
            } else if strings.EqualFold(config.CalendarValue004, e.GetSummary()) {
                fmt.Println("LoadICAL() - Current even is", e.GetSummary(), "-", &e)
                return 4
            } else if strings.EqualFold(config.CalendarValue005, e.GetSummary()) {
                fmt.Println("LoadICAL() - Current even is", e.GetSummary(), "-", &e)
                return 5
            }
        }
    }

    return 0
}

Perfect, now we have the summary of the current active event, we have checked that against the config to see what entry it refers to, then reported back an integer number that references it. This can now be passed to a function to set the server state accordingly.

func main() {
    // READ command-line arguments to get config file path
    // The [1:] skips the first word, the name of the binary
    CliArguments := os.Args[1:]

    // Pass first CLI argument as config file location
    // LoadConfiguration then returns the data into the Config struct
    config := LoadConfiguration(CliArguments[0])

    // LoadICAL and return current event summary ON/OFF
    // This will return an integer number, which I correlate with the config for the appropriate request payload
    // This part is effectively limitless in the number of options you can have
    // We are passing in the config struct, so that the function can see that data
    desiredStatus := LoadICAL(config)

    // Send request to API with desiredStatus
    SetServerState(config, desiredStatus)
}
// SetServerState power state from desired state
func SetServerState(config Config, state int) {
    // First we check to see if we should ignore self signed certificate warnings
    // This is configured in the config file
    if config.ServerAPIIgnoreCertError == true {
        // CHANGE SECURITY TO IGNORE SELF SIGNED CERTIFICATE WARNINGS
        http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
    
    // We pull down the url here, just as a test of what we've changed above
    // If there is an error since this change, we dump the error to StdOut
        _, err := http.Get("https://golang.org/")
        if err != nil {
            fmt.Println(err)
        }
    }

    // PERFORM REQUEST ON API
    // Set payload to match desired state
    // We use the value we got in the previous function and map this to the payload value in the config file
    // Again this is not a scalable element, if doing a lot more than 5 options it will start to get very big
    var payload = strings.NewReader("")
    if state == 0 {
        payload = strings.NewReader(config.ServerAPIPayloadDefault)
    } else if state == 1 {
        payload = strings.NewReader(config.ServerAPIPayload001)
    } else if state == 2 {
        payload = strings.NewReader(config.ServerAPIPayload002)
    } else if state == 3 {
        payload = strings.NewReader(config.ServerAPIPayload003)
    } else if state == 4 {
        payload = strings.NewReader(config.ServerAPIPayload004)
    } else if state == 5 {
        payload = strings.NewReader(config.ServerAPIPayload005)
    }

    // Build & trigger the request
    req, _ := http.NewRequest("POST", config.ServerAPIPath, payload)
    req.SetBasicAuth(config.ServerAPIUser, config.ServerAPIPass)
    req.Header.Add("Cache-Control", "no-cache")
    res, _ := http.DefaultClient.Do(req)

    defer res.Body.Close()
    body, _ := ioutil.ReadAll(res.Body)
    // Print the result of request
    fmt.Println(res)
    fmt.Println(string(body))
}

GoLang Compile

During initial testing I installed GoLang tools onto the Pi and ran the code through that, having Go compile at runtime. However, this was noticeably slow. On my desktop the difference is barely noticeable, the code isn’t super long nor complex, but there is a slight delay. On the Pi however, it was multiple seconds per run, which is not what I was aiming for. Although I wasn’t going to be sat watching it run, I always had the intension of compiling the code, and this additional delay during testing confirmed to me the power of compiling for the platform. Once I started to compile the ARM binary, the execution time was instant, as on my desktop.

To compile for the Pi, the GoLang environment variables specify the environment to which to compile for, you can see your current ones with go env. To change these depends on your system, Linux or Windows, Google is your friend here. For the Pi, running Raspbian, you will want to have the following set. If instead you are using a BSD base OS, such as RaspBSD, your variables will be different. GoLang supports many different platforms, you will have to know the intended platform and specify for this.

GOOS=linux 
GOARCH=arm 
GOARM=5

Once compiled, you can push the binary to the Pi with SCP (or USB, or SD, etc). Secure Copy (SCP) is included in most if not all Linux OSs, tools can be downloaded for Windows too.

scp ./main.go pi@192.168.1.10:/home/pi/apiResourceScheduler.arm.bin
scp ./config.json pi@192.168.1.10:/home/pi/config.json

Raspbian & Cron

As mentioned, Raspbian was used as the OS for the RasbperryPi, as recommended by the Pi team. Rasbian is based on Debian, so feels very much as you would expect, with some nice little features added in by the Pi team.

A favourite of mine, is being able to add in a 'ssh' file to root directory, that is an empty file called 'ssh', to [enable headless ssh for the Pi](https://www.raspberrypi.org/documentation/remote-access/ssh/). As someone who likes to tidy cables away, pulling out the HDMI for the Pi is a pain, this little thing is great.

The code above works, but at present needs to be run manually, not much of a scheduler at this point. The code could have been coded to wait for certain times to trigger an action, but this isn’t really reliable as then needs to be running 24 /7, nor recommended. Linux already has a scheduling functionality built in, called Cron Jobs, this is the recommended approach. I utilised that to run the code at a regular interval. On Rasbian the users cron jobs can be configured with crontab -e. This command will open the default text editor with the configuration loaded, all you have to do is add your own entries.

# Edit this file to introduce tasks to be run by cron.
#
# <truncated output>
#
# m h  dom mon dow   command
#
# Run apiRequestScheduler every 1 minute
# Output the StdOut to a logfile
*/1 * * * * /home/pi/apiRequestScheduler/apiRequestScheduler.arm.bin /home/pi/apiRequestScheduler/config.json 1>> /home/pi/apiRequestScheduler/apiRequestScheduler.log
# Clear apiRequestScheduler log every 1st of the month
# I don't want the SD card to get full at any point
* * 1 * * echo "Log Cleared Daily from 'crontab -e'" > /home/pi/server_control/server_control.log
# Clear apiRequestScheduler temporary files every 1st of the month
# I don't want the SD card to get full at any point
* * 1 * * rm -rf /home/pi/server_control/tmp/*

The re-occurring frequency is handled by the prefix of numbers, these relate to minutes, hours, day, month, year. To have an entry repeat every minute, as used by the example above, ‘step values’ can be utilised. This is to say, in the minute column, a value of */1 will run every minute. A value of */2 will run every 2 minutes, etc. Whereas a value of 30 will run on the 30th minute. This is fairly configurable once you get your head around it, the crontab documentation has some good examples of these, but for this project every minute meets the requirement.

Every minute the shell runs the command /home/pi/apiRequestScheduler/apiRequestScheduler.arm.bin /home/pi/apiRequestScheduler/config.json 1>> /home/pi/apiRequestScheduler/apiRequestScheduler.log which triggers the binary, passes in the config file and lastly outputs StdOut to a log file. Initially I had used a local config file reference ./config.json in the code. However, the cron job is not run from the local directory of the binary, so the config is not in ./. I updated the code to take in a config path, which can be specified in this command.

I do prefer this approach, as it allows others to use a different path if they prefer.

Closing Statements

That’s it, to see it all as one please look here on GitHub - main.go. There is not a lot of complexity in what goes on, thanks to GoLang, PuloV and the RaspberryPi team it was fun to setup and get a working tool in the end. That’s not to say I didn’t use Google and StackOverflow constantly. I feel the mix of different technologies was the right approach, if not at least an interesting endeavour for myself, as it allows me to use the best of different tool-set and get something really easy, quickly out the door, with good amount of flexibility.