Introducing the Constrained Application Protocol (CoAP) for Java

Use CoAP for lightweight messaging applications, such as for working with simple Internet of Things sensors or other devices.

The Constrained Application Protocol (CoAP) was created for smaller, constrained devices that run on low-powered networks with possibly transient or lossy connectivity. CoAP is similar to HTTP or REST communication in that the messages generally fall into the categories of GET, POST, PUT, and DELETE.

This article covers the basics of CoAP and how to program it in Java server and client applications. A follow-up article will explore advanced topics such as the Observer pattern, device discovery, and cross-protocol proxies.

Introducing CoAP

The CoAP specification is maintained by the Internet Engineering Task Force (IETF). CoAP is very conversational by nature because it is request- and response-driven. Contrast that to the publish/subscribe design of the MQTT protocol.

At a lower level, CoAP messages are sent and received over User Datagram Protocol (UDP), which by nature is unreliable, so a basic reliability scheme is built into CoAP on top of UDP. For added security, messages can be sent using the Datagram Transport Layer Security (DTLS) protocol instead of UDP. In either case, each CoAP message needs to fit into a single UDP/DTLS datagram packet. Further, CoAP supports datagram messaging over IPv4 and IPv6 networks and variants such as 6LoWPAN , as shown in Figure 1.

Figure 1. CoAP messaging network communication layers

Although unicast UDP is used for request- and response-driven CoAP, multicast UDP messaging is used to support CoAP device/sensor discovery. CoAP clients and servers support a special “all CoAP nodes” multicast address, with port 5683, to discover other CoAP servers and their shared resources.

Exploring CoAP

First, this article explores various aspects of the protocol, and then it presents Java code to send and process CoAP messages.

The CoAP message model. All message exchanges in CoAP are like those for HTTP. With CoAP, all interchanges are asynchronous and datagram-based. Optional reliability is built into the message exchange using a timeout and retransmission protocol based on random and increasing back-off timers with eventual timeout. Figure 2 shows CoAP’s two-layer approach to messaging.

Figure 2. The CoAP two-layer messaging approach

CoAP messages include a four-byte fixed-length header. Depending upon the message type, this header is followed by optional header data and the payload. Each message includes a 16-bit message ID used to link requests to their accompanying acknowledgments (ACKs) or error statuses (when applicable). A nonconfirmable (NON) message (that is, one that doesn’t require confirmation or, hence, reliability) is sent from the client to the server with no ACK (see Figure 3).

Figure 3. A NON request message and response

If the recipient is unable to process the message, it may reply with a reset message (RST) to indicate this. An example of when high reliability may not be needed is with telemetry data, because there is a constant stream of updates so missing one update isn’t devastating.

In short, message IDs are unique values that are used to identify duplicate messages and match confirmable messages (CONs) to ACKs. Tokens are unique values used to match requests to responses.

For example, client 1 sends a nonconfirmable GET request for a temperature reading to client 2. A unique message ID (0x101) and token (0x21) are provided. The message ID is useful to detect message duplication (more on that later). For a request/response message exchange, the token must match across all associated messages. Therefore, in this example, the message IDs will be unique for both the request and response messages, but the token (0x21) will be the same for both.

In the case of a NON message, either the request or the response may be lost. For some types of message exchanges, this may be acceptable. For cases where that is not acceptable, CoAP supports CONs, as shown in Figure 4.

In Figure 4, client 1 sends a confirmable GET request for a temperature reading to client 2, providing a unique message ID (0x101) and token (0x21). The token must match throughout the entire message exchange. When client 2 receives the CON request, it sends an ACK message containing the same message ID provided with the request.

Figure 4. A CON request message and response

Later, client 2 sends a CON response message with the requested data and a new message ID, using the same token as in the CON request from client 1. To confirm that it received the data, client 1 sends an ACK to client 2 containing the same message ID as in the response (0x92ab), and the exchange is complete.

Piggybacking. To reduce message traffic and processing overhead, CoAP supports the concept of piggybacking. With this, the response data to a request can be included (piggybacked) on the ACK message; see Figure 5.

Figure 5. CON response content piggybacked onto ACK messages

The CON request is identical to the previous exchange. However, when client 2 sends the ACK, it also includes the response data. When client 1 receives this ACK with data, the CON message exchange is complete.

Handling lost messages. CoAP specifies a way to detect lost UDP messages and then retransmit them. For instance, in any of the CON examples above, if an ACK for the initial CON request was not received, after a timeout period the CON request will be resent with the identical message ID and token, as shown in Figure 6.

Figure 6. Lost CON request message resent later

In the case where the original CON request arrived at client 2 but the CON ACK was lost instead, after a timeout client 1 will resend the original request, as shown in Figure 7.

Figure 7. CON request arrives, but ACK is lost

For a scenario where piggybacking is not used and an explicit CON response is sent, the failure scenario may be slightly different, as Figure 8 shows.

Figure 8. A CON request is sent, and then an ACK is sent. The response arrives, but the final ACK is lost.

In this scenario, the CON request, its ACK, and its CON response are all received. However, client 1’s ACK back to client 2 (which would otherwise complete the exchange) is lost. After a timeout, client 2 assumes its CON response was lost and resends it, which client 1 receives and detects as a duplicate. Client 1 ignores the response but resends the ACK, completing the CON message exchange.

The message retransmission timeout interval. When any CON message is sent, an ACK must be received within a random time interval that is somewhere between the ACK_TIMEOUT and a value calculated as (ACK_TIMEOUT * ACK_RANDOM_FACTOR).

If an ACK is not received in time, the sender retransmits the CON message at exponentially increasing intervals until it receives an ACK (or an RST message) or it runs out of the number of attempts defined by the MAX_RETRANSMIT value. Each time a message is retransmitted, that message’s transmit counter is incremented and its wait time is doubled.

The messaging discussed so far has been viewed from the unicast UDP perspective. However, CoAP supports multicast messaging. In this paradigm, requests can be sent to groups of servers reachable at a UDP multicast address, where all or some of the servers listening may respond to a single request.

CoAP URI and security overview. The CoAP uniform resource identifier (URI) uses coap:// (for communication over UDP) and coaps:// (for communication over DTLS) to locate resources. The resources are organized hierarchically. The format is as follows:

◉ URI = "coap://<host>[:<port>]<path>[?<query>]"

◉ URI = "coaps://<host>[:<port>]<path>[?<query>]"

Here are two examples.

◉ coap://example.com:5683/~sensors/readings.xml

◉ coap://FF05::FD:5683/.well-known/core

CoAP, by default, is not secure and works in what is called NoSec mode, in which no security, encryption, or authentication is performed. However, the specification covers many security points, and here are a few.

◉ Because CoAP is a subset of HTTP/S, many HTTP/S security considerations apply to CoAP also.

◉ For secure communications, CoAP messaging can be performed over DTLS instead of UDP.

◉ CoAP provides support for key-based certificates and access control lists for authentication and authorization.

◉ To avoid URI parsing vulnerabilities, CoAP reduces the scope for parsing and defines a concise range of encodable values with reduced complexity and risk.

◉ CoAP provides guidance on proxies and caching to avoid or at least raise awareness of the risks of “in-the-middle” attacks.

◉ CoAP provides strategies for avoiding request amplification, such as CoAP server slicing and blocking modes.

◉ CoAP detects address spoofing and encourages limited response rates.

CoAP method definitions and Java code

CoAP messaging is analogous to HTTP communication, but it’s not equivalent. For example, both CoAP and HTTP support the same four basic commands—GET, POST, PUT, and DELETE—but the semantics of the commands vary slightly. For instance, many of the success and error codes are different between the two.

CoAP GET. As is the case with REST, this method retrieves a representation of the information that corresponds to the resource within the URI at the time the request is made. As is the case with HTTP, the request can include an Accept option that suggests the preferred content of the response.

The GET response status codes are 2.03 for a valid request, 2.05 for a valid request with content, and 4.05 for “not allowed.” Listing 1 is a CoAP GET from a Java client using the Californium CoAP library.

Listing 1.  Java code to make a CoAP GET for temperature data

public class CoapGetClient {

   public static void main(String args[])

         throws URISyntaxException {

       // make synchronous get call

       URI uri = new URI("coap://192.168.1.97:5683/temp");

       CoapClient client = new CoapClient(uri);

       CoapResponse response = client.get();

       if ( response != null ) {

         byte[] bytes = response.getPayload();

         System.out.println(response.getCode());

         System.out.println(response.getOptions());

         System.out.println(response.getResponseText());

         System.out.println("\nDETAILED RESPONSE:");

         System.out.println(Utils.prettyPrint(response));

       }

   }

}

Thanks to the Californium library implementation, the call to CoapClient.get() is synchronous and will wait to return a code and payload if it’s successful. You can use response.getPayload() to retrieve the payload as a byte array   or use the getResponseText() shortcut, as shown in this example. Also shown is the call to retrieve the return code and any optional parameters.

Finally, the Californium library provides a utility class with a prettyPrint() method, which formats the response details as shown in Listing 2. You can download the synchronous CoAP GET application from my GitHub repository.

Listing 2.  The output of the CoAP GET request for the current temperature, with complete response details

2.05

{"Content-Format":"text/plain"}

70 degrees F

DETAILED RESPONSE:

==[ CoAP Response ]============================================

MID    : 53522

Token  : d09b7c5ede

Type   : ACK

Status : 2.05 - CONTENT

Options: {"Content-Format":"text/plain"}

RTT    : 10 ms

Payload: 12 Bytes

---------------------------------------------------------------

70 degrees F

A return code of 2.05 indicates that the response was sent as part of the CoAP ACK message. Also, the payload in the response to the /temp request was the 70 degrees F text. If an error had occurred instead, the return code would have indicated the error and no payload would have been provided.

To contrast, the code in Listing 3 is for an asynchronous CoAP GET request, where execution continues after the request is sent. The response will be processed sometime later in the supplied callback. You can download the complete asynchronous get application from my GitHub repository.

Listing 3. An asynchronous CoAP GET request, with a callback to handle the response

class AsynchListener implements CoapHandler {

   @Override

   public void onLoad(CoapResponse response) {

       String content = response.getResponseText();

       System.out.println("onLoad: " + content);

   }

@Override

   public void onError() {

       System.err.println("Error");

   }

}

// ...

AsynchListener asynchListener = new AsynchListener();

client.get( asynchListener );

After the GET request is made, the response will arrive asynchronously via the onload method within the AsynchListener class, which extends the Californium library’s CoapHandler callback class.

The CoAP server code (see Listing 4) that runs on the temperature device, or the gateway connected to the temperature sensor, handles the GET request.

Listing 4.  A CoAP listener that responds to requests for the current temperature

import org.eclipse.californium.core.CoapServer;

...

public class CoapTempServer extends CoapServer {

    private static final int COAP_PORT = 

            Configuration.getStandard().get( CoapConfig.COAP_PORT );

    private static final String tempUnti = "F";

    float temperature = 70;

    public static void main(String[] args) {

        try {

            CoapTempServer server = new CoapTempServer();

            server.start();

        }

        catch ( Exception e ) {

            System.err.println("CoAP server err: " + e.getMessage());

        }

    }

    public CoapTempServer() throws SocketException {

        super();

        addEndpoints();

        add( new TemperatureResource() );

    }

    private void addEndpoints() {

        Configuration config = Configuration.getStandard();

        // Add an endpoint listener for each host network interface

        for (InetAddress addr : 

             NetworkInterfacesUtil.getNetworkInterfaces()) {

            InetSocketAddress bindToAddress = 

                new InetSocketAddress(addr, COAP_PORT);

            CoapEndpoint.Builder builder = new CoapEndpoint.Builder();

            builder.setInetSocketAddress(bindToAddress);

            builder.setConfiguration(config);

            addEndpoint( builder.build() );

        }

    }

    class TemperatureResource extends CoapResource {

        public HelloWorldResource() {

            super("temp"); // set resource URI identifier

            getAttributes().setTitle("Server room temperature");

        }

        @Override

        public void handleGET(CoapExchange exchange) {

            // get latest temperature reading and return it

            temperature = …

            exchange.respond(temperature + " degrees " + tempUnit);

        }

    }

}

The call to addEndpoints in the constructor sets up the socket binding and listening for each host network address. Next, the CoAP resource, implemented in the nested class TemperatureResource, is created and added to the CoAP server. This code gets mapped to the associated URI. Finally, the call to server.start is all it takes to listen for requests, which are routed to the TemperatureResource.handleRequest method.

If, for example, you run this code on a gateway server connected to multiple devices or sensors, you can add additional CoapResource implementations, each mapped to unique URI identifiers, to return different data, such as humidity data; see Listing 5. You can download the complete application from my GitHub repository.

Listing 5. Adding multiple CoapResource implementations to one CoAP server

public class CoapGetServer extends CoapServer {

    // ...

    public CoapGetServer() throws SocketException {

        super();

        addEndpoints();

        add(new TemperatureResource());

        add(new HumidityResource());

    }

    class HumidityResource extends CoapResource {

        public HumidityResource() {

            super("humidity"); // set resource identifier

            getAttributes().setTitle("Server room humidity");

        }

        @Override

        public void handleGET(CoapExchange exchange) {

            exchange.respond(humidity + " percent");

        }

    }

    class TemperatureResource extends CoapResource {

        // ...

    }

}

With GET requests, the data flows in one direction. Next, let’s explore how POST requests make things more interesting.

CoAP POST. A CoAP POST request asks the recipient to process the representation (data) enclosed within the request. The actual function performed by the POST is dependent on the target. It typically results in the target resource being updated or a new resource being created if it doesn’t yet exist. If it’s created, the response should include the new URI for it.

Listing 6 shows two CoAP POST requests being made to the CoAP server resource named data: one to supply plain text and another to supply XML. You can download the complete client application from my GitHub repository.

Listing 6. A CoAP POST request is like GET but with form data added.

CoapClient client = new CoapClient("coap://192.168.1.10:5683/data");

CoapResponse r1 = client.post("data", MediaTypeRegistry.TEXT_PLAIN);

CoapResponse r2 = client.post( "<data>this is data</data>", 

                               MediaTypeRegistry.APPLICATION_XML );

In this example, requests to the CoAP server resource named data are processed in the handlePOST method (see Listing 7) of the CoapResource interface implementation.

Listing 7.  Handling CoAP POST requests

public class MyCoapServer extends CoapResource {

   public static void main(String[] args) {

       CoapServer server = new CoapServer();

       server.add(new MyCoapServer("data"));

       server.start();

   }

   @Override

   public void handlePOST(CoapExchange exchange) {

       exchange.accept();

       int format = exchange.getRequestOptions()

                            .getContentFormat();

       if (format == MediaTypeRegistry.APPLICATION_XML) {

           String xml = exchange.getRequestText();

           String responseTxt = "Received XML: '" + xml + "'";

           System.out.println(responseTxt);

           exchange.respond(CREATED, responseTxt);

       }

       else if (format == MediaTypeRegistry.TEXT_PLAIN) {

           // ...

           String plain = exchange.getRequestText();

           String responseTxt = "Received text: '" + plain + "'";

           System.out.println(responseTxt);

           exchange.respond(CREATED, responseTxt );

       }

       else {

           // ...

           byte[] bytes = exchange.getRequestPayload();

           System.out.println("Received bytes: " + bytes);

           exchange.respond(CREATED);

       }

   }

   // ...

}

The handlePOST method can be written to handle specific data types, such as plain text, XML, JSON, images, and other types. You can download the complete application from my GitHub repository.

CoAP PUT and DELETE. The PUT method is like POST with a subtle difference: If the resource doesn’t exist, the target has the option to create the resource. The DELETE method requests that the resource identified by the target resource URI be deleted. The DELETE status codes are 2.02 and 4.05.

Source: oracle.com

Continue reading

Java vs. Python: A comparison of machine learning libraries

A close look at the performance of Python’s scikit-learn vs. Java’s Tribuo

Machine learning (ML) is important because it can derive insights and make predictions using an appropriate dataset. As the amount of data being generated increases globally, so do the potential applications of ML. Specific ML algorithms can be difficult to implement, since doing so requires significant theoretical and practical expertise.

Fortunately, many of the most useful ML algorithms have already been implemented and are bundled together into packages called libraries. The best libraries for performing ML need to be identified and studied, since there are many libraries currently available.

Scikit-learn is a very well-established Python ML library widely used in industry. Tribuo is a recently open sourced Java ML library from Oracle. At first glance, Tribuo provides many important tools for ML, but there is limited published research studying its performance.

This project compares the scikit-learn library for Python and the Tribuo library for Java. The focus of this comparison is on the ML tasks of classification, regression, and clustering. This includes evaluating the results from training and testing several different models for each task.

This study showed that the new Tribuo ML library is a viable, competitive offering and should certainly be considered when ML solutions in Java are implemented.

This article explains the methodology of this work; describes the experiments which compare the two libraries; discusses the results of the experiments and other findings; and, finally, presents the conclusions. This article assumes readers have familiarity with ML’s goals and terminology.

Methodology

To make a comparison between scikit-learn and Tribuo, the tasks of classification, regression, and clustering were considered. While each task was unique, a common methodology was applicable to each task. The flowchart shown in Figure 1 illustrates the methodology followed in this work.

Figure 1. A flowchart that illustrates the methodology of this work

Identifying a dataset appropriate to the task was the logical first step. The dataset needed to be not too small, since a model should be trained with a sufficient amount of data. The dataset also needed to be not too large, to allow the models being developed to be trained in a reasonable amount of time. The dataset also needed to possess features which could be used without requiring excessive preprocessing.

This work focused on the comparison of two ML libraries, not on the preprocessing of data. With that said, it is almost always the case that a dataset will need some preprocessing.

The data preprocessing steps were completed using Jupyter notebooks, entirely in Python and occasionally using scikit-learn’s preprocessing functionality. Any required cleanup, scaling, and one-hot encoding was done during this step. Fortunately, Sebastian Raschka and Vahid Mirjalili, in Python Machine Learning (third edition), provide several clear examples of when these types of changes to data are required.

Once the data preprocessing was complete, the data was re-exported to a comma-separated value formatted file. Having a single, preprocessed dataset facilitated the comparison of a specific ML task between the two libraries by isolating the training, testing, and evaluation of the algorithms. For example, the classification algorithms from the Tribuo Java library and the scikit-learn Python library could load exactly the same data file. This aspect of the experiments was controlled very carefully.

Choosing comparable algorithms. To make an accurate comparison between scikit-learn and Tribuo, it was important that the same algorithms were compared. This third step of defining the common algorithms for each library required studying each library to identify what algorithms are available that could be accurately compared. For example, for the clustering task, Tribuo currently only supports K-Means and K-Means++, so these were the only algorithms common to both libraries which could be compared. Furthermore, it was critical that each algorithm’s parameters were precisely controlled for each library’s specific implementation.

To continue with the clustering example, when the K-Means++ object for each library was instantiated, the following parameters were used:

◉ maximum iterations = 100

◉ number of clusters = 6

◉ number of processors to use = 4

◉ deterministic randomness for centroids = 1

The next step was to identify each library’s best algorithm for a specific ML task. This involved testing and tuning several different algorithms and their parameters to see which one performed the best. For some people, this is when ML is really fun!

Concretely, for the regression task, the random forest regressor and XGBoost regressor were found to be the best for scikit-learn and Tribuo, respectively. Being the best in this context meant to achieve the best score for the task’s evaluation metric. The process of selecting the optimal set of parameters for a learning algorithm is known as hyperparameter optimization.

A side note: In recent years, automated machine learning (AutoML) has emerged as a way to save time and effort in the process of hyperparameter optimization. AutoML is also capable of performing model selection, so in theory this entire process could be automated. However, the investigation and use of AutoML tools was out of the scope of this work.

Evaluating the algorithms. Once the preprocessed datasets were available, the libraries’ common algorithms had been defined, and the libraries’ best scoring algorithms had been identified, it was time to carefully evaluate the algorithms. This involved verifying that each library split the dataset into training and test data in an identical way but was only applicable to the classification and regression tasks. This also required writing some evaluation functions which produced similar output for both Python and Java, and for each of the ML tasks.

At this point, it should be clear that Jupyter notebooks were used for these comparisons. Because there were two libraries and three ML tasks, six different notebooks were evaluated: A single notebook was used to perform the training and testing of the algorithms for each ML task for one of the libraries. From a terminology standpoint, a notebook is also referred to as an experiment in this work.

Throughout this study many, many executions of each notebook were performed for testing, tuning, and so forth. Once everything was finalized, three independent executions of each experiment were made in a very controlled way. This meant that the test system was running only the essential applications, to ensure that a maximum amount of CPU and memory resources were available to the experiment being executed. The results were recorded directly in the notebooks.

The final step in the methodology of this work was to compare the results. This included calculating the average training times for each model’s three training times. The average of each algorithm’s three recorded evaluation metrics was also calculated. These results were all reviewed to ensure the values were consistent and no obvious reporting error had been made.

Experiments

The three ML tasks of classification, regression, and clustering are described in this section. Note that the version of scikit-learn used was 0.24.1 with Python 3.9.1. The version of Tribuo was 4.1 with Java 12.0.2. All of the experiments, the datasets, and the preprocessing notebooks are available for review online in my GitHub repository.

Classification. The classification task of this work focused on predicting if it would rain the next day, based on a set of weather observations collected for the current day. The Kaggle dataset used was Rain in Australia. The dataset contained 140,787 records after preprocessing, where each record was a detailed set of weather information such as the current day’s minimum temperature and wind information. To clean up the data, features with large numbers of missing values were removed. Features which were categorical were one-hot encoded and numeric features were scaled.

Three classification algorithms common to each library were compared: stochastic gradient descent, logistic regression, and decision tree classifier. The algorithm which obtained the best score for the scikit-learn library using this data was the multi-layer perceptron. For Tribuo, the best scoring algorithm was the XGBoost classifier.

F1 scores were used to compare each algorithm’s ability to make correct predictions. It was the best metric to use since the dataset was unbalanced. In the test data, of a total of 28,158 records, there were 21,918 recordings with no rain and only 6,240 entries indicating rain.

Regression. The regression task studied here used a dataset containing the attributes of a used car to predict its sale price. The Kaggle dataset used was Used-cars-catalog, and some examples of the car attributes were mileage, year, make, and model. The preprocessing effort for this dataset was minimal. Some features were dropped, and some records with empty values were removed. Only three columns were one-hot encoded. The resulting dataset contained 38,521 records.

Like the classification task described above, there were three algorithms common to scikit-learn and Tribuo which were compared: stochastic gradient descent, linear regression, and decision tree regressor. The scikit-learn library’s algorithm that achieved the best score was the random forest regressor. The best scoring algorithm for the Tribuo library was the XGBoost regressor. The values of root mean square error (RMSE) and R2 score were used to evaluate the algorithms. These are common metrics used to evaluate regression tasks.

Something else is worth mentioning: In this experiment for the Tribuo library, sometimes loading the preprocessed dataset took a very long time. This issue has been fixed in the 4.2 release of Tribuo. Although data loading is not the focus of this study, poor data loading performance can be a significant problem when ML algorithms are used.

Clustering. The clustering task used a generated dataset of isotropic Gaussian blobs, which are simply sets of points normally distributed around a defined number of centroids. This dataset used six centroids. In this case, each point had five dimensions. There were a total of six million records in this dataset, making it quite large.

The benefit of using an artificial dataset like this is that a point’s assigned cluster is known, which is useful for evaluating the quality of the clustering. Otherwise, evaluating a clustering task is more difficult. Using the cluster assignments, an adjusted mutual information score is used to evaluate the clusters, which indicates the amount of correlation between the clusters.

There are only two clustering algorithms currently implemented in the Tribuo library. They are K-Means and K-Means++. Since these algorithms are also available in scikit-learn, they could be compared. Other clustering algorithms from the scikit-learn library were tested to see if a better- or equal-scoring model could be identified. Surprisingly, there does not seem to be any other scikit-learn algorithm which can complete training within a reasonable amount of time using this large dataset.

Experimental results

Here are the results for classification, regression, and clustering.

Classification. As mentioned above, the F1 score is the best way to evaluate these classification algorithms, and an F1 score close to 1 is better than a score not close to 1. The F1 “Yes” and “No” scores for each class were included. It was important to observe both values since this dataset was unbalanced. The results in Table 1 show that the F1 scores were very close for the algorithms common to both libraries, but Tribuo’s models were slightly better.

Table 1. Classifier results using the same algorithm

Table 2 indicates that Tribuo’s XGBoost classifier obtained the best F1 scores out of all the classifiers in a reasonable amount of time. The time values used in the tables containing these results are always the algorithm training times. This work was interested in the model which achieved the best F1 scores, but there could be other situations or applications which are more concerned with model training speed and have more tolerance for incorrect predictions. For those cases, it is worth noting that the scikit-learn training times are better than those obtained by Tribuo—for the algorithms common to both libraries.

Table 2. Classifier best algorithm results

It is helpful to have a visualization focusing on these F1 scores. Figure 2 shows a stacked column chart combining each model’s F1 score for the Yes class and the No class. Again, this shows Tribuo’s XGBoost classifier model was the best.

Figure 2. A stacked column chart combining each model’s F1 scores

Regression. Keep in mind that a lower RMSE value is a better score than a higher RMSE value, and the R2 score closest to 1 is best.

Table 3 shows the results from the regression algorithms common to the scikit-learn and Tribuo libraries. Both libraries’ implementations of stochastic gradient descent scored very poorly, so those huge values are not included here. Of the remaining algorithms common to both libraries, scikit-learn’s linear regression model scored better than Tribuo’s linear regression model, and Tribuo’s decision tree model beat out scikit-learn’s model.

Table 3. Regressor results using the same algorithm

Table 4 shows the results for the model from each library which produced the lowest RMSE value and highest R2 score. Here, the Tribuo XGBoost regressor model achieved the best scores, which were just slightly better than the scikit-learn random forest regressor.

Table 4. Regressor results for the best algorithm

Visualizations of these tables, which summarize the scores from the regression experiments, reinforce the results. Figure 3 shows a clustered column chart of the RMSE values, while Figure 4 shows a clustered column chart of the R2 scores. The poor scoring stochastic gradient descent models are not included. Recall that the two columns on the right are comparing each library’s best scoring model, which is why the scikit-learn random forest model is side by side with the Tribuo XGBoost model.

Figure 3. A clustered column chart comparing the RMSE values

Figure 4. A clustered column chart comparing the R2 scores

Clustering. For a clustering model, an adjusted mutual information value of 1 indicates perfect correlation between clusters.

Table 5 shows the results of the two libraries’ K-Means and K-Means++ algorithms. It is not surprising that most of the models get a 1 for their adjusted mutual information value. This is a result of how the points in this dataset were generated. Only the Tribuo K-Means implementation did not achieve a perfect adjusted mutual information value. It’s worth mentioning again that although there are several other clustering algorithms available in scikit-learn, none of them could finish training using this large dataset.

Table 5. Clustering results

Additional findings

Comparing library documentation. To prepare the ML models for comparison, the scikit-learn documentation was heavily consulted. The scikit-learn documentation is outstanding. The API docs are complete and verbose, and they provide simple, relevant examples. There is also a user guide which provides additional information beyond what’s contained in the API docs. It is easy to find the appropriate information when models are being built and tested.

Good documentation is one of the main goals of the scikit-learn project. At the time of writing, Tribuo does not have an equivalent set of published documentation. The Tribuo API docs are complete and there are helpful tutorials which describe how to perform the standard ML tasks. To perform tasks beyond this requires more effort, but some hints can be found by reviewing the appropriate unit tests in the source code.

Reproducibility. There are certain situations when ML models are used where reproducibility is important. Reproducibility means being able to train or use a model repeatedly and observe the same result with a fixed dataset. This can be difficult to achieve, for example, when a model depends on a random number generator and the model has been trained several times causing several invocations of the model’s random number generator.

Tribuo provides a feature called Provenance, which is ubiquitous throughout the library’s code. Provenance captures the details on how any dataset, model, etc. is created and has been modified in Tribuo. This information would include the number of times a model’s random number generator has been used. The main benefit it offers is that any of these objects can be regenerated from scratch, assuming the original training and testing data are used. Clearly this is valuable for reproducibility. Scikit-learn does not have a feature like Tribuo’s Provenance feature.

Other considerations. The comparisons described in this work were done using Jupyter notebooks. It is well known that Jupyter includes a Python kernel by default. However, Jupyter does not natively support Java. Fortunately, a Java kernel can be added to Jupyter using a project called IJava. The functionality provided by this kernel enabled the comparisons made in this study. Clearly, these kernels are not directly related to the libraries under study but are noted since they provided the environment in which these libraries were exercised.

The usual comment that Python is more concise than Java wasn’t really applicable in these experiments. The var keyword, which was introduced in Java 10, provides local variable type inference and reduces some boilerplate code often associated with Java. Developing functions in the notebooks still requires defining the types of the parameters since Java is statically typed. In some cases, getting the generics right requires referencing the Tribuo API docs.

Earlier, it was mentioned that the data preprocessing steps were completed entirely in Python. It is significantly easier to perform data preprocessing or data cleaning activities in Python, compared to Java, for several reasons. The primary reason is the availability of supporting libraries which offer rich data preprocessing features, such as pandas. The quality of a dataset being used to build an ML model is so important; therefore, the ease with which data preprocessing can be performed is an important consideration.

The ML tasks of classification, regression, and clustering were the focus of the comparisons made in this work. It should be noted again that scikit-learn provides many more algorithm implementations than Tribuo for each of these tasks. Furthermore, scikit-learn offers a broader range of features, such as an API for visualizations and dimensionality reduction techniques.

Source: oracle.com

Continue reading

Pattern matching updates for Java 19’s JEP 427: when and null

The third preview of pattern matching for switch addresses case refinement and the proper handling of null cases.

This article looks at changes to pattern matching for switch in its third preview: case refinement and handling null cases. This discussion assumes that you already know how pattern matching for switch works in the second preview, which is Java 18’s JEP 420.

The third-preview changes, targeted for Java 19, are described in JEP 427. Note that these are nitty-gritty details, and they may very well change before pattern matching for switch becomes finalized in a future iteration of the platform.

The big picture is that switch is gaining expressiveness and is becoming a more relevant programming construct in Java. Of course, while switch is an important place to use patterns, it won’t be the only place.

That means that pattern matching functionality is evolving to handle larger considerations such as integrating how these additions work with the more familiar current understanding of switch. It’s important that the semantics of patterns used in switch align with semantics used elsewhere.

Case refinement

The classic switch statement compares a variable to specific values and picks the branch that matches.

With patterns in switch, the variable is matched against types. Each type is essentially a big bag of all the values that are legal for that type. For example, the Integer type includes all integers between about minus 2 billion and about plus 2 billion. The String type includes all character strings.

However, you will not always want to treat all instances of a type exactly the same. You may wish to distinguish between positive and negative integers, or you might want to consider strings that do or don’t contain a specific substring.

These conditions can of course easily be expressed with an if, but in the context of a switch, an if would require a type pattern, an arrow (->), and then the if before, finally, the statements you are actually interested in.

For example, instead of condition, arrow, statements you would get condition-part-one, arrow, condition-part-two, statements, as follows:

switch (object) {

  case Integer i ->

    if (i >= 0)

      // positive integers

    else

      // negative integers

  case String s ->

    if (s.contains("foo"))

      // strings with "foo"

    else

      // strings without "foo"

  default -> // ...

}

This is where guarded patterns come in, or rather, came in. You see, JEP 427 proposes the use of when clauses instead. Both allow adding Boolean conditions to a pattern to identify the desired case, such as positive integers, on the left side and then putting simple statements after the arrow, for example,

switch (object) {

  case Integer i ____ i >= 0 -> // positive integers

  case Integer i -> // negative integers

  default -> // ...

}

(The ____ is a placeholder for something coming later in this article.)

The when clause, as proposed by JEP 427, differs from guarded patterns in the following two aspects:

◉ Which construct owns the refinement

◉ How that refinement is expressed

As the name suggests, guarded patterns were part of the pattern syntax, which was very powerful. For example, once nested patterns were introduced, you could add Boolean conditions inside a large pattern, not just at the end.

Overall, guarded patterns had some weird edge cases, though, that the JDK team wants to avoid. So, now it’s no longer the pattern that owns a refinement. Now, the case owns the refinement, as shown below.

switch (shape) {

  // now-obsolete guarded patterns with

  // record patterns from JEP 405

  case Point(int x && x > 0, int y) -> // use positive x, y

  default -> // ...

}

switch (shape) {

  // refinement owned by 'case' can't be "inside" the pattern

  case Point(int x, int y) ____ x > 0 -> // use positive x, y

  default -> // ...

}

The other aspect is the syntax of how to express a refinement. You may be used to seeing && as a strongly binding operator between equitable terms. This notation worked reasonably well for guarded patterns because they were actually part of the patterns, but it works less well if case owns the refinement.

As Brian Goetz wrote on the Project Amber mailing list, “It’s harder to imagine && as part of the case, and not as part of the pattern.”

Thus, the current proposal is to use the new context-specific keyword when between the pattern and the refining Boolean conditions, and that’s what goes into the ____ placeholder shown earlier.

switch (object) {

  case Integer i when i >= 0 ->

    // positive integers

  case Integer i ->

    // negative integers

  default -> // ...

}

Null values

My favorite topic to rant about is null! Once again, it sullies beauty with its dark presence, specifically, because it’s necessary to deal with null in a pattern switch.

Historically, switch simply throws a NullPointerException when the variable is null. However, the more you use switch over complicated types, the more urgent becomes the need to find a better way to handle that situation than a separate if before the switch.

Ever since the first preview version of pattern matching for switch (unchanged by JEP 427), it has been possible to add a case null for this special situation and even combine that with a default. What happens without that case?

String string = // ???

// JDK 18 and JEP 427

switch (string) {

  case null -> // ...

  case "foo" -> // ...

  case "bar" -> // ...

}

In the second preview in JDK 18, the answer to that depends on the presence of an unconditional pattern, that is, a pattern that matches all possible instances of the switched variable’s type.

Think of a switch over a variable of type Shape where the last case is case Shape s. That code always matches; it’s unconditional on type Shape.

Unconditional patterns even match null. Therefore, in JDK 18 the variable s could be null. However, that would probably lead to several NullPointerException situations, and I wasn’t a fan of silently sweeping null in with the other shapes, as shown below.

Shape shape = // ...

// as previewed in JDK 18

switch (shape) {

  case Point p -> ...

  // unconditional pattern

  //  ~> matches 'null'

  //  ~> 's' can be 'null'

  case Shape s -> ...

}

Fortunately, JEP 427 proposes to change that unpleasant situation. How? Unconditional patterns still match null, but switch won’t let it get that far. If there’s no case null, switch throws a NullPointerException without even looking at the patterns.

Shape shape = // ...

// as proposed by JEP 427:

// no 'case null' ~> NPE

switch (shape) {

  case Point p -> ...

  // unconditional pattern

  //  (still matches 'null')

  case Shape s -> ...

}

Interestingly, this top-level behavior does not extend to nested patterns, though.

An unconditional nested pattern will still match null, which introduces a sharp edge during refactoring. This is inconsistent, but consistently not matching null also has weird effects, such as not being able to write a single pattern that matches all instances of a record, as shown below.

interface Shape { }

record Circle(Point center)

  implements Shape { }

// JEP 427 + JEP 405

Shape shape = // ...

switch (shape) {

  // 'Point center' is unconditional

  // the circle's center 'Point'

  case Circle(Point center) ->

    // 'center' may be 'null'

  case Shape s ->

    // 's' won’t be 'null'

}

A solution to this kerfuffle that I’ll personally be pursuing is to flat-out avoid null. (Actually, I’m already doing that, but that is beside the point.)

Anyway, when null isn’t legal, switch doesn’t have to mention it, and while I would’ve found it nice if switch threw exceptions upon encountering null, it isn’t really a switch’s job to do that.

Source: oracle.com

Continue reading

The Case Against Logging

The one thing that all enterprise applications have in common, regardless of their framework, language, or communication protocols is that they produce logs, logs that are allegedly human-readable lines of strings that aim to help us to debug, trace, or otherwise inspect what’s going on. The question is, how useful are these logs and do they help us solve problems that actually happen. In this post, I will explore this question, highlight some of the challenges that come with logging, and explore alternative solutions.

Historically, logging had always been a part of enterprise software. We’ve all seen a lot of logging frameworks and may even have created our own. There are lots of conversations about supposedly best practices on how to implement reasonable logs. If you ask developers, logging can be used for debugging, tracing, journaling, monitoring, and printing errors. In general, every piece of information that developers think may be important will be written to a log.

When I refer to logging in this post, it relates to writing (more or less) human-readable strings to files or to stdout, similar to:

2022-02-14 07:10:25.800 [http-nio-8080-exec-7] My custom log format INFO  com.sebastian_daschner.example.CarOrderController - User info@example.com ordered car with color: red, manufacturer: BMW, model: M3

2022-02-14 07:09:25.915 [http-nio-8080-exec-37] My custom log format INFO  com.sebastian_daschner.example.HelloController - /hello called with param World, for the 535th time

2022-02-14 07:09:26.817 [http-nio-8080-exec-5] My custom log format INFO  com.sebastian_daschner.example.CarOrderController - User test@example.com ordered car with color: blue, manufacturer: Audi, model: A3

...

You might ask: Sebastian, what exactly is wrong with that?

Shortcomings

When it comes to the kind of logging and typical JVM logging frameworks (Log4j, Slf4j, Logback, etc.) I’m describing in this post, we can identify certain issues:

Performance is certainly the biggest one. If you talk to a JVM performance expert, they will tell you that how you log can have one of the biggest, if not the biggest, negative impact on your application’s performance. If you really want your application to perform poorly, you should add a lot of logging statements in your main use cases. This logging should engage in creating and concatenating a lot of strings. And no logging framework is without several layers of buffering. The biggest performance issue for applications is the cost of a Java heap allocation, and logging usually allocates disproportionately when compared to typical business code.

It’s not just allocation costs as high allocation rates ultimately will hyper-activate the garbage collector. This in turn will result in high CPU utilization and increased frequency of tail latency. It’s quite interesting to have a look at such scenarios in production, in which an application allegedly utilizes the CPU a lot, which in fact turns out to be caused by the garbage collector because of excessive allocation.

Disk I/O is another factor to consider. Writing and flushing a lot of data to disk will impact the performance of all applications running on the same hardware. Even worse, log files that reside in network storage impact the throughput even more, since the write operation hits the operating system I/O twice, with file system and network calls involved. Logging makes these hardware devices which are shared between all applications part of the critical path. We often see this as a “noisy neighbor”.

The number of dependencies that logging frameworks bring along, directly or transitively, creates a few potential issues, as well. Not only do dependencies and their transitive baggage inflate the application’s size and build time. The more dependencies and moving parts we have, the higher the changes that there are version conflicts, bugs, or security issues, which not least the recent Log4Shell vulnerability has shown once again. In general, the less dependencies we include the better.

Log formats and which one to choose are another issue. Which information should be include (I dare you to have a look at the Log4j layouts), how should we structure our logging messages, which information should be logged at which log level, and so on. On the one hand, logs are produced to be human-readable but the volume of data the logs lead to creates the necessity to use more tooling such as log aggregators and search engines. Unfortunately, human-readable formats tend to be expensive for machines to read which leads to the conclusion that logs are generally neither really human nor machine readable.

In this cases, it makes more sense to consider a binary or a more machine-friendly format. Some applications do log lines of JSON, but the question remains, is this really any better? The log format will impact performance not only with regards to how much is added to each line, but also how many string concatenations, method invocations, and reflection lookups have to be performed.

Log levels are another topic that I haven’t seen being used reasonably out there in real-world projects, which I think is not the projects’ fault. Log levels might sound nice in theory, that you observe different levels of detail as to what information is currently interesting, but in practice that doesn’t really work well. Once some certain condition happened that you would have liked to debug using log statements, it’s likely that the log level wasn’t detailed enough, since debug and trace logs are usually not available in production. After all, keeping detailed log levels on in production that result in many statements being written to disk will hurt your performance. Some implementations support the ability to dynamically change the log level at runtime. The only issue is that by the time you realize you need more information, it’s likely too late to capture what is needed. Choosing a proper log level, and then of course which statements should be logged in the code using which level, is thus always a trade-off. Often this task is left to the developers to decide without any architectural guidance and this further degrades the usefulness of the logs.

Logs can produce huge amounts of data that is written to log files and over time can result in large files that are cumbersome to handle and expensive to parse. Parsing log formats in general bears quite an avoidable overhead. Especially if we collect all logs in another stack such as ELK, these tools then need to parse all these formats just again, which makes one question if the approach was a sensible one to begin with.

In general, one might doubt if logging is the best choice for implementing debugging, tracing, journaling, monitoring, or printing errors. If this is the case, then what are the alternatives? Lets start this discussion by looking at why we log.

Why developers use logging

Developers have different reasons why they use logging in an application running in production. At first, let’s look into negative examples — concerns that should be implemented differently:

◉ Debugging (appending debug statements into the log)

◉ Journaling (writing business-related events or audits, usually synchronously)

◉ Tracing (printing method invocations, arguments, and stacks)

◉ Monitoring (appending business or technical metrics to the log)

◉ Health checks (writing status messages that ensure the application is still alive)

Using logging for debugging in production does not only have a huge negative performance impact but also might not even be of much help. Information that isn’t available at a configured log level won’t help you to reproduce a specific bug and setting a log level that is too low, especially for third-party framework code and libraries, typically results in an overwhelming number of lines being logged per user request. Debugging race conditions and concurrency-related errors will very likely change the race which will lead to a different outcome and again be of little help. When debugging functionality, it’s more advisable to use a debugger, such as the one that is included in your IDE, that can connect against a running application, either locally or remotely.

Logging statements that aim to record business-motivated information in order to create an audit train is akin to a poor man’s version of journaling. This activity is better accomplished by using a journaling solution or event sourcing. If the business requirements demand a journaling, audit log, or alike, it should be treated as such and made explicit. Using event sourcing or proper journaling technology such as Chronicle Queue persists the messages more efficiently, with lower footprint, lower latency, and higher throughput.

Business- or technically-motivated tracing should also be made explicit in the application and implemented using a fit-for-purpose solution, such as OpenTracing or another journaling solution. (Mis-)using logging in order to trace method invocations and arguments has the same drawbacks as using logging for debugging.

Instead of using logging to monitor, one should use a proper monitoring solution, which usually includes emitting metrics in an endpoint. For example, publishing metrics using the Prometheus format via HTTP, and then scraping those metrics at the receiving end. All major enterprise frameworks support solutions out of the box, and even custom-built implementations are usually a better choice for realizing monitoring than appending individual metrics to the log.

The same holds true for health checks, for which the applications should offer some sort of health checking functionality like an endpoint or status page. These solutions integrate nicely into containerized environments like Kubernetes.

When and how to do logging

Now, we’re seen many cases for which we should avoid using traditional logging — so should we even use logging and if so when?

A great usage of stdout logging is for debugging while in development mode, such as the Quarkus dev mode. I very often insert a quick System.out.println, reload the application which in case of Quarkus takes 1-2 seconds, see the result, adapt my code, and remove the logging statement usually right away. This is often faster than starting up the debug mode of the IDE, at least for simple debugging. However, one should not leave these statements in the final committed version that runs in production.

Logging is useful for showing the completion of automated jobs, that otherwise won’t easily be seen. Single lines in the log that summarize the outcome of the job that has been executed can turn out very helpful, if they don’t appear often, that is if the job runs rarely, ideally every other minute or less.

It is helpful to log errors that are unrecoverable or unexpected especially when all other means to expose this information have failed. For example, if your application is unable to connect to a database, logging maybe the only way to capture the error information. You may want to suppress this error in cases where multiple user invocation will cause a multitude of retries. In this case, we’re not really interested in the performance impact on the “broken” application but more in being a good citizen by minimizing the impact on the rest of the system.

It’s also helpful to log any error that you didn’t expect or account for, such as the ones that hint to a bug. I know, you might ask: “what error is expected, anyways?”. Take NullPointerException as an example. This error implies a faulty, unexpected condition that the code is not handling properly. This is different from a user-provoked error which usually shouldn’t end up in the log. For example, I was once debugging my router when it showed a NullPointerException stack trace in the web interface. It turned out, the code was not properly handling the condition when there were no cables connected. This is a user-provoked condition that wouldn’t require further developer intervention yet instead of signaling that a cable needed to be connected, I was instead presented with this technical exception that had no useful information. User-provoked does include technical users and external systems, such as failures that are caused during a HTTP client request. That’s the distinction I’d make: You can log any exception that implies that the developers need to investigate and fix something.

Containers and orchestration frameworks have had some impact in as to how logging should be done. Rotating log files aren’t required anymore and container frameworks typically take the container log stream, that is the stdout and stderr, and correlate it further. For that reason, what might sound very controversial to a few of you, if I do have to log, I use these wonderful loggers that have been shipped with Java since JDK 1.0, called System.out and System.err. To some of you this might sound overly simplistic or even ignorant, but quite the contrary: using a simple, straightforward logging solution avoids a multitude of potential transitive problems.

The impact and transitive dependencies that third-party libraries have on our applications are not to be neglected. Unless we have a specific requirement, it does make sense to go with a simple, straightforward solution, to which we comprehend what it does, its consequences and benefits.

Source: javacodegeeks.com

Continue reading