domingo, 25 de janeiro de 2015

SCD4J: Continuous Delivery has never been so simple

Simple Continuous Delivery for Java and Groovy Developer (SCD4J) is an automation platform for configuring and installing your Web IT infrastructure. With just one command you can install clusters and deploy applications in any stage of your software development process. Note that SCD4J is not another option for Puppet or Chef. It is, actually, a simpler option than those tools.

The main advantages of using SCD4J over its competitors are:

  • Designed with a very specific purpose in mind
    • Small and medium enterprises which have opted in to use Java technologies and Continuous Delivery
  • No environment configuration
    • No need to install and configure coordinator servers
    • No need to install agents on clients
  • Small learning curve for Java and Groovy Developers
  • Built upon modern and well known Java technologies
  • Smooth integration with your existing binary repositories (i.e. Artifactory, Nexus, etc)
  • Very simple architectural model
    • Use module and files hooks to implement the automation logic
  • Developers get closer to sysadmins once they can quickly and easily jump into the infrastructure coding

Infrastructure Automation


After the 2010’s Continuous Delivery boom, too much has been talked about how to automate builds in order to anticipate issues, to assemble higher quality software and to ensure the creation of more reliable applications. However, fewer resources on the internet focus on the infrastructure automation.

It is important to say that one of the main issues I’ve seen during all the time I’ve worked as a Software Consultant is the lack of infrastructure standards (mainly the lack of standards between different stages of the software development process) and the error prone manual configurations performed by sysadmins. Both of these problems are well addressed using an automation infrastructure tool. Unfortunately, current well known solutions are too complex for non programming sysadmins and have a huge learning curve for Java developers. Making this kind of adoption, in rushy companies, somewhat difficult.

That is where SCD4J shines. Its simple architecture and specific purpose design makes it a very good match for pairing the knowledge of sysadmins and Java developers to build your company's Web infrastructure quickly.

SCD4J: creating a new project


To start a new project, first we must create a file called build.gradle, as show in the below example:

plugins {
   id "com.datamaio.scd4j" version "0.7.0"
}

scd4j {
   install {
       modules "my_first_module"         // should be any dir into modules dir
       config     "my_first_config.conf"   // should be any properties file into config dir
   }
}

Then, we may run the task newproject of our build.gradle using the installed gradle build tool.

Note: We need to have Gradle 2.1 (or higher) installed just for the first run. After that, SCD4J will automatically install a Gradle wrapper. Please, see Requirements for more details.

Once the execution has finished we will have the following directory structure created:

build.gradle
config/
     my_first_config.conf
modules/
     my_first_module/
           Module.hook
gradlew
gradlew.bat
gradle/...

The config dir contains the configuration files. In our case the my_first_config.conf file (i.e. a Java properties file) in which we must put all the variables that will be used by our modules.

The modules dir contains a new module called my_first_module. This is the place where we will implement our automation. Note that the newproject task also created a file called Module.hook, which we are going to talk a little bit more later in this article.

Finally, we can see a gradle directory and gradlew and gradlew.bat files. Those are the gradle wrapper and, from now one, you can pack this project and run it in any machine without needing to install gradle.


SCD4J: basics

Similarly to any other infrastructure automation tool you can think about SCD4J as a “smart copy tool”. In which we should add files using the full path of the target environment. For example, suppose we would like to install a file called standalone.xml into the directory /opt/wildfly/standalone/configuration, then we should simply add this file in our my_first_module as shown in the below example:

build.gradle
config/
     my_first_config.conf
modules/
     my_first_module/
           /opt/wildfly/standalone/configuration/standalone.xml
...

Running again our project, we can notice that SCD4J has simply copied our file to the right folder. But, in real situations, we want to install slightly different standalone.xml files depending on the environment we are running on. For example, we don’t want to mess with our production database. So, if we are installing the testing environment we need to set a given database URL and password, similarly, if we are installing a development environment we need to set another URL and another password. In order accomplish this we must put those two variable in the my_first_config.conf file and tell SCD4J that this file is a template. For that, we just need append .tmpl at the end of the file name.

build.gradle
config/
     my_first_config.conf
modules/
     my_first_module/
           /opt/wildfly/standalone/configuration/standalone.xml.tmpl
...

This way, SCD4J understands that this file will need to be modified during runtime and then we can use one of the existing template options (Groovy Template is the default) to resolve the logic you would like to implement.

It is important to say that for each environment we run our project on we must provide different config files. In order to understand how to do that, take a look at packaging and distribution session of our documentation

Ok... But what about more complex installations?!? Lets say we would like to dynamically generate http port number based on the server and environment we are running on. For those kind of logic, SCD4J offers .hook files.

build.gradle
config/
     my_first_config.conf
modules/
     my_first_module/
           /opt/wildfly/standalone/configuration/standalone.xml.tmpl
           /opt/wildfly/standalone/configuration/standalone.xml.tmpl.hook
...

As we can see in the above example, we need to create a file with exactly the same name of the original one appending .hook at end. Note that this file is a groovy script in which you can define pre and post implementations. For example:

pre {
   // write here the code to execute before the file installation.
   def httpPort = dynamicallyBuildHttpPort()
   set(“port”, httpPort)      // The port variable will be accessible in the post and in the .tmpl file
}
post {
  // write here the code to execute after the file installation.
  // usually changes file permission, changes file owner, rename it, etc
}

The same way we can define a hook for a file, we can also define a hook for a module. For that, we need to create a file named as Module.hook in my_first_module dir. It is important to say that the pre definition must return a constant CONTINUE or ABORT. If nothing is returned, CONTINUE is assumed (in <FileName>.hook the constant SKIP_FILE is also allowed). Take the bellow code as an example:

pre{
   // put here your pre module installation logic
   if( isLinux() ) {
       return CONTINUE
   }
   return ABORT
}

post{
   // put here your post module installation logic
}

The above example is saying that the module will be executed only on Linux boxes. If we run this module against any other operational system, the installation will be aborted.

It is important to say that SCD4J provides many helpful functions which can be used in the Module.hook and any <FileName>.hook. Those functions allow us to install packages, change file permissions, start/stop a service, run OS specific commands, and so one. To have a complete understanding of which helpers we can use into hooks, take a look at ModuleHook and FileHook public APIs, respectively.

Conclusion


Before finish this short introduction to SCD4J we would like to highlight the core benefiting of using it to install and configure your Web infrastructure:

  • Manage IT risks in a simple and efficient way
  • Repeatability
  • People's time is precious, so use it for smarter tasks
  • Spread delivery  knowledge
  • Infrastructure fully documented
  • Documentation will never get out of date

We strongly suggest you to take a look at the project documentation and examples. There you will find more complex options and examples (such as the installation of a wildfly cluster). So that you can quickly learn how to implement real world infrastructure automation.

Thanks for the interest of making the software delivery somewhat more professional. Hope you enjoy our tool.

Sincerely, SCD4J Team


Short Story

Some years ago I was called to help a big company to improve their software quality. Shortly I discovery that the main problem was not the software itself but the process to develop and delivery it. These issues led them to a lot of difficulties to put new versions in production. That situation has generated a real war between IT and Development teams, which, of course, tried to blame on each other. It took me some years to convince those managers to make sysadmins and developers work all together. Recently, more than one year have passed, I accidentally met one of the integration guys who told me that the last time they had an issue in production was before using our tools. That is one of the reasons why I choose to rewrite and publish it as a free software. Maybe other companies may get such benefits from it.

quinta-feira, 19 de dezembro de 2013

Using JAX-RS 2.0 to build an app from scratch.


Hi every one,

I would like to share with my friends my last article about JAX-RS 2.0. This article has been published this month (December) in the Brazilian Java Magazine and it shows, with a practical example, how to develop a full app using the new stuff of this spec.

Unfortunately, this time I could not share its content, but for those who live in Brazil I would recommend to buy one magazine in the nearest newsstand. :-). For those who live outside of Brazil I can only share the source code with you  :-(



source code: https://github.com/fbrubbo/BrazilianJavaMagazine

Anyway.. Thanks for reading.

domingo, 11 de agosto de 2013

Running Websockets on OpenShift




WebSocket is a pretty new and important technology in which it is possible to send message frames from server to clients without opening multiple HTTP connections. Although it is has been proven to be a solid and consolidated specification, running websockets in the existing cloud platforms is still a pain in the ass. The main cloud players do not provide enough support for WebSocket protocol due to the lack of application servers’ readiness and due to other issues related to the routing layer.

Knowing those difficulties, I’ve decide to post the steps to help eager people to use Red Hat OpenShift PaaS to run a websocket apps in the cloud. So, let's start from the beginning.


  • First of all, you must sign up in the OpenShift web site
  • Then you must add a new application
    • Pick Do-It-Yourself (DIY) cartridge
      • This will allow you to use a “pure Linux server” to do whatever you would like to do. In the specific case of this post, we are going to install an embedded web application in order to provide the required support for websockets.
    • Once DIY cartridge have been selected, you must fill out the application name and click on ‘Create Application’ button
    • Finally, do a ‘git clone’ according instructions

At this point, we have already prepared our Linux server on the cloud to install our application according our needs. Now, let’s make the required changes in the working copy we have just cloned from OpenShift master repository.

The changes are:
  • Update the shell script file which will be used by OpenShift to start our app
    • File: .openshift/action_hoooks/start 
    • Content: 

#!/bin/bash
# The logic to start up your application should be put in this
# script. The application will work only if it binds to
# $OPENSHIFT_DIY_IP:8080


echo "Starting WebSocketExample............. $OPENSHIFT_DIY_IP:$OPENSHIFT_DIY_PORT"
nohup java -jar $OPENSHIFT_REPO_DIR/diy/WebSocketExample.war > $OPENSHIFT_DIY_DIR/logs/server.log 2>&1 &
echo "Started WebSocketExample............. $OPENSHIFT_DIY_IP:$OPENSHIFT_DIY_PORT"
  • Update the shell script file which will be used by OpenShift to stop your app
    • File: .openshift/action_hoooks/stop 
    • Content:

#!/bin/bash
source $OPENSHIFT_CARTRIDGE_SDK_BASH


echo "Stopping WebSocketExample............."
if [ -z "$(ps -ef | grep WebSocketExample.war | grep -v grep)" ]
then
   client_result "WebSocketExample Application is already stopped"
else
echo "Killing WebSocketExample............."
   kill `ps -ef | grep WebSocketExample.war | grep -v grep | awk '{ print $2 }'` > /dev/null 2>&1
fi
echo "Stopped WebSocketExample............."
  • Add the embedded jetty application in the ‘Do It Yourself’ directory
    • Directory: diy
      • In this example, we have used the ‘WebSocketExample.war’ app name in order to match out our start/stop scripts.
      • Important note about the embedded jetty application file: 
        • There are some peculiaridades in the implementation of the class com.embedded.JettyStarter (shown below in red) which must be changed in order to bind the embedded app to the right IP and PORT number in the OpenShift platform. 
        • Content:

public class JettyStarter {
       public static void main(String[] args) throws Exception{
               ProtectionDomain domain = JettyStarter.class.getProtectionDomain();
               URL location = domain.getCodeSource().getLocation();
       
               // create a web app and configure it to the root context of the server
               WebAppContext webapp = new WebAppContext();
               webapp.setDescriptor("WEB-INF/web.xml");
               webapp.setConfigurations(new Configuration[]{ new AnnotationConfiguration()
        , new WebXmlConfiguration(), new WebInfConfiguration(), new MetaInfConfiguration()        
        //, new FragmentConfiguration(), new EnvConfiguration(), new PlusConfiguration()
                });
               webapp.setContextPath("/");
               webapp.setWar(location.toExternalForm());
       
               // starts the embedded server and bind it to openshift variables        
               String host = System.getenv("OPENSHIFT_DIY_IP");
               String port = System.getenv("OPENSHIFT_DIY_PORT");
               System.out.println(host + ":" + port);
               InetSocketAddress sa = new InetSocketAddress(getByName(host), valueOf(port));
               Server server = new Server(sa);


               server.setHandler(webapp);        
               server.start();
               server.join();
       }
}

After script files have been configured and the embedded application has been put in the ‘diy’ directory, you must commit and push changes back to the master repository.
  • The commands are:
    • git commit -m 'My changes'
    • git push
  • As soon as the push finishes, the stop/start scripts will run
    • If you notice it didn’t run accordingly, remote access your openshift server and check if everything is ok. Remember: it is Red Hat Linux, so you should be ok checking your stuff.
    • Try to check if start/stop scripts have execution permission (not sure why, but I have this issue in some experiments).
      • ls -lt $OPENSHIFT_REPO_DIR/.openshift/action_hooks/
    Finally, are done. \o/
      Access 'ws://[YOUR APP].rhcloud.com:8000/example' using your favorite WebSocket tool and check it by yourself.
        Don't have a WebSocket tool?  Take a look on ‘Advanced REST client’ for Chrome

        The openshift configuration used in this example can be found here.
          Thanks for reading