Monthly Archives: August 2016

10 Essential WordPress Interview Questions


  1. Consider the following code snippet. Briefly explain what changes it will achieve, who can and cannot view its effects, and at what URL WordPress will make it available.
add_action('admin_menu', 'custom_menu');

function custom_menu(){
    add_menu_page('Custom Menu', 'Custom Menu', 'manage_options', 'custom-menu-slug', 'custom_menu_page_display');
}

function custom_menu_page_display(){
    echo '<h1>Hello World</h1>';
    echo '<p>This is a custom page</p>';
}

With default settings and roles, admins can view it and all lower roles can’t. In fact this menu item will only be visible to users who have the privilege to “manage options” or change settings from WordPress admin dashboard.

The admin custom page will be made available at this (relative) URL: “?page=custom-menu-slug”.

2. How would you change all the occurrences of “Hello” into “Good Morning” in post/page contents, when viewed before 11AM?

In a plugin or in theme functions file, we must create a function that takes text as input, changes it as needed, and returns it. This function must be added as a filter for “the_content”.

It’s important that we put a little effort to address some details:

  • Only change when we have the full isolate substring “hello”. This will prevent words like “Schellong” from becoming “sgood morningng”. To do that we must use “word boundary” anchors in regular expression, putting the word between a pair of “\b”.
  • Keep consistency with the letter case. An easy way to do that is to make the replace case sensitive.
<?php
function replace_hello($the_content){
    if(current_time('G')<=10){
        $the_content=preg_replace('/\bhello\b/','good morning',$the_content);
        $the_content=preg_replace('/\bHello\b/','Good Morning',$the_content);
    }
    return $the_content;
}
add_filter('the_content', 'replace_hello');

3. What is the $wpdb variable in WordPress, and how can you use it to improve the following code?

<?php
function perform_database_action(){
    mysql_query(“INSERT into table_name (col1, col2, col3) VALUES ('$value1','$value2', '$value3');
}

$wpdb is a global variable that contains the WordPress database object. It can be used to perform custom database actions on the WordPress database. It provides the safest means for interacting with the WordPress database.

The code above doesn’t follow WordPress best practices which strongly discourages the use of any mysql_query call. WordPress provides easier and safer solutions through $wpdb.

The above code can be modified to be as follows:

<?php
function perform_database_action(){
    global $wpdb;
    $data= array('col1'=>$value1,'col2'=>$value2,'col3'=>$value3);
    $format = array('%s','%s','%s');
    $wpdb->insert('table_name', $data, $format);
}
add_custom_script();
function add_custom_script(){
    wp_enqueue_script( 
        'jquery-custom-script',
        plugin_dir_url( __FILE__ ).'js/jquery-custom-script.js'
    );
}

wp_enqueue_script is usually used to inject javascript files in HTML.

The script we are trying to queue will not be added, because “add_custom_script()” is called with no hooks. To make this work properly we must use the wp_enqueue_scripts hook. Some other hooks will also work such as init, wp_print_scripts, and wp_head.

Furthermore, since the script seems to be dependent on jQuery, it’s recommended to declare it as such by adding array(‘jquery’) as the 3rd parameter.

Proper use:

add_action(‘wp_enqueue_scripts’, ‘add_custom_script’);
function add_custom_script(){
    wp_enqueue_script( 
        'jquery-custom-script',
        plugin_dir_url( __FILE__ ).'js/jquery-custom-script.js',
        array( 'jquery')
    );
}

5. Assuming we have a file named “wp-content/plugins/hello-world.php” with the following content. What is this missing to be called a plugin and run properly?

<?php
add_filter('the_content', 'hello_world');
function hello_world($content){
    return $content . "<h1> Hello World </h1>";
}

The file is missing the plugin headers. Every plugin should include at least the plugin name in the header with the following syntax:

<?php
/*
Plugin Name: My hello world plugin
*/

6. What is a potential problem in the following snippet of code from a WordPress theme file named “footer.php”?

...
        </section><!—end of body section- ->
        <footer>All rights reserved</footer>
    </body>
</html>

All footer files must call the <?php wp_footer() ?> function, ideally right before the </body> tag. This will insert references to all scripts and stylesheets that have been added by plugins, themes, and WordPress itself to the footer.

7. What is this code for? How can the end user use it?

function new_shortcode($atts, $content = null) {
    extract(shortcode_atts(array(
        “type” => “warning”
    ), $atts));
    return '
'.$content.'
'; } add_shortcode(“warning_box”, “new_shortcode”);

This shortcode allows authors to show an info box in posts or pages where the shortcode itself is added. The HTML code generated is a div with a class name “alert” plus an extra class name by default, “alert-warning”. A parameter can change this second class to change the visual aspect of the alert box.

Those class naming structures are compatible with Bootstrap.

To use this shortcode, the user has to insert the following code within the body of a post or a page:

[warning_box]Warning message[/warning_box]

8. Is WordPress safe from brute force login attempts? If not, how can you prevent such an attack vector?

No, WordPress on its own is vulnerable to brute force login attempts.

Some good examples of actions performed to protect a WordPress installation against brute force are:

  • Do not use the “admin” username, and use strong passwords.
  • Password protect “wp-login.php”.
  • Set up some server-side protections (IP-based restrictions, firewall, Apache/Nginx modules, etc.)
  • Install a plugin to add a captcha, or limit login attempts.

9. The following line is in a function inside a theme’s “function.php” file. What is wrong with this line of code?

wp_enqueue_script('custom-script', '/js/functions.js');

Assuming that “functions.js” file is in the theme’s “js/” folder, we should use ‘get_template_directory_uri()’. '/js/functions.js' or the visitors’ browser will look for the file in the root directory of the website.

10. Suppose you have a non-WordPress PHP website with a WordPress instance in the “/blog/” folder. How can you show a list of the last 3 posts in your non-WordPress pages?

One obvious way is to download, parse, and cache the blog’s RSS feeds. However, since the blog and the website are on the same server, you can use all the WordPress power, even outside it.

The first thing to do is to include the “wp-load.php” file. After which you will be able to perform any WP_Query and use any WordPress function such as get_posts, wp_get_recent_posts, query_posts, and so on.

<?php
    require_once('../blog/wp-load.php');
?>
<h2>Recent Posts</h2>
<ul>
<?php
    $recent_posts = wp_get_recent_posts(array(‘numberposts’=>3));
    foreach($recent_posts as $recent){
        echo '<li><a href="' . get_permalink($recent["ID"]) . '">' . $recent["post_title"] . '</a></li> ';
    }
?>
</ul>

Source: Toptal

Writing Tests That Matter: Tackle The Most Complex Code First


There are a lot of discussions, articles, and blogs around the topic of code quality. People say – use Test Driven techniques! Tests are a “must have” to start any refactoring! That’s all cool, but it’s 2016 and there is a massive volume of products and code bases still in production that were created ten, fifteen, or even twenty years ago. It’s no secret that a lot of them have legacy code with low test coverage.

While I’d like to be always at the leading, or even bleeding, edge of the technology world – engaged with new cool projects and technologies – unfortunately it’s not always possible and often I have to deal with old systems. I like to say that when you develop from scratch, you act as a creator, mastering new matter. But when you’re working on legacy code, you’re more like a surgeon – you know how the system works in general, but you never know for sure whether the patient will survive your “operation”. And since it’s legacy code, there are not many up to date tests for you to rely on. This means that very frequently one of the very first steps is to cover it with tests. More precisely, not merely to provide coverage, but to develop a test coverage strategy.

Coupling and Cyclomatic Complexity: Metrics for Smarter Test Coverage

Forget 100% coverage. Test smarter by identifying classes that are more likely to break.

Basically, what I needed to determine was what parts (classes / packages) of the system we needed to cover with tests in the first place, where we needed unit tests, where integration tests would be more helpful etc. There are admittedly many ways to approach this type of analysis and the one that I’ve used may not be the best, but it’s kind of an automatic approach. Once my approach is implemented, it takes minimal time to actually do the analysis itself and, what is more important, it brings some fun into legacy code analysis.

The main idea here is to analyse two metrics – coupling (i.e., afferent coupling, or CA) and complexity (i.e. cyclomatic complexity).

The first one measures how many classes use our class, so it basically tells us how close a particular class is to the heart of the system; the more classes there are that use our class, the more important it is to cover it with tests.

On the other hand, if a class is very simple (e.g. contains only constants), then even if it’s used by many other parts of the system, it’s not nearly as important to create a test for. Here is where the second metric can help. If a class contains a lot of logic, the Cyclomatic complexity will be high.

The same logic can also be applied in reverse; i.e., even if a class is not used by many classes and represents just one particular use case, it still makes sense to cover it with tests if its internal logic is complex.

There is one caveat though: let’s say we have two classes – one with the CA 100 and complexity 2 and the other one with the CA 60 and complexity 20. Even though the sum of the metrics is higher for the first one we should definitely cover the second one first. This is because the first class is being used by a lot of other classes, but is not very complex. On the other hand, the second class is also being used by a lot of other classes but is relatively more complex than the first class.

To summarize: we need to identify classes with high CA and Cyclomatic complexity. In mathematical terms, a fitness function is needed that can be used as a rating – f(CA,Complexity) – whose values increase along with CA and Complexity.

Generally speaking, the classes with the smallest differences between the two metrics should be given the highest priority for test coverage.

Finding tools to calculate CA and Complexity for the whole code base, and provide a simple way to extract this information in CSV format, proved to be a challenge. During my search, I came across two tools that are free so it would be unfair not to mention them:

A Bit Of Math

The main problem here is that we have two criteria – CA and Cyclomatic complexity – so we need to combine them and convert into one scalar value. If we had a slightly different task – e.g., to find a class with the worst combination of our criteria – we would have a classical multi-objective optimization problem:

We would need to find a point on the so called Pareto front (red in the picture above). What is interesting about the Pareto set is that every point in the set is a solution to the optimization task. Whenever we move along the red line we need to make a compromise between our criteria – if one gets better the other one gets worse. This is called Scalarization and the final result depends on how we do it.

There are a lot of techniques that we can use here. Each has its own pros and cons. However, the most popular ones are linear scalarizing and the one based on an reference point. Linear is the easiest one. Our fitness function will look like a linear combination of CA and Complexity:

f(CA, Complexity) = A×CA + B×Complexity

where A and B are some coefficients.

The point which represents a solution to our optimization problem will lie on the line (blue in the picture below). More precisely, it will be at the intersection of the blue line and red Pareto front. Our original problem is not exactly an optimization problem. Rather, we need to create a ranking function. Let’s consider two values of our ranking function, basically two values in our Rank column:

R1 = A∗CA + B∗Complexity and R2 = A∗CA + B∗Complexity

Both of the formulas written above are equations of lines, moreover these lines are parallel. Taking more rank values into consideration we’ll get more lines and therefore more points where the Pareto line intersects with the (dotted) blue lines. These points will be classes corresponding to a particular rank value.

Unfortunately, there is an issue with this approach. For any line (Rank value), we’ll have points with very small CA and very big Complexity (and visa versa) lying on it. This immediately puts points with a big difference between metric values in the top of the list which is exactly what we wanted to avoid.

The other way to do the scalarizing is based on the reference point. Reference point is a point with the maximum values of both criteria:

(max(CA), max(Complexity))

The fitness function will be the distance between the Reference point and the data points:

f(CA,Complexity) = √((CA−CA )2 + (Complexity−Complexity)2)

We can think about this fitness function as a circle with the center at the reference point. The radius in this case is the value of the Rank. The solution to the optimization problem will be the point where the circle touches the Pareto front. The solution to the original problem will be sets of points corresponding to the different circle radii as shown in the following picture (parts of circles for different ranks are shown as dotted blue curves):

This approach deals better with extreme values but there are still two issues: First – I’d like to have more points near the reference points to better overcome the problem that we’ve faced with linear combination. Second – CA and Cyclomatic complexity are inherently different and have different values set, so we need to normalize them (e.g. so that all the values of both metrics would be from 1 to 100).

Here is a small trick that we can apply to solve the first issue – instead of looking at the CA and Cyclomatic Complexity, we can look at their inverted values. The reference point in this case will be (0,0). To solve the second issue, we can just normalize metrics using minimum value. Here is how it looks:

Inverted and normalized complexity – NormComplexity:

(1 + min(Complexity)) / (1 + Complexity)∗100

Inverted and normalized CA – NormCA:

(1 + min(CA)) / (1+CA)∗100

Note: I added 1 to make sure that there is no division by 0.

The following picture shows a plot with the inverted values:

Final Ranking

We are now coming to the last step – calculating the rank. As mentioned, I’m using the reference point method, so the only thing that we need to do is to calculate the length of the vector, normalize it, and make it ascend with the importance of a unit test creation for a class. Here is the final formula:

Rank(NormComplexity , NormCA) = 100 − √(NormComplexity2 + NormCA2) / √2

More Statistics

There is one more thought that I’d like to add, but let’s first have a look at some statistics. Here is a histogram of the Coupling metrics:

What is interesting about this picture is the number of classes with low CA (0-2). Classes with CA 0 are either not used at all or are top level services. These represent API endpoints, so it’s fine that we have a lot of them. But classes with CA 1 are the ones that are directly used by the endpoints and we have more of these classes than endpoints. What does this mean from architecture / design perspective?

In general, it means that we have a kind of script oriented approach – we script every business case separately (we can’t really reuse the code as business cases are too diverse). If that is the case, then it’s definitely a code smell and we need to do refactoring. Otherwise, it means the cohesion of our system is low, in which case we also need refactoring, but architectural refactoring this time.

Additional useful information we can get from the histogram above is that we can completely filter out classes with low coupling (CA in {0,1}) from the list of the classes eligible for coverage with unit tests. The same classes, though, are good candidates for the integration / functional tests.

You can find all the scripts and resources that I have used in this GitHub repository: ashalitkin/code-base-stats.

Does It Always Work?

Not necessarily. First of all it’s all about static analysis, not runtime. If a class is linked from many other classes it can be a sign that it’s heavily used, but it’s not always true. For example, we don’t know whether the functionality is really heavily used by end users. Second, if the design and the quality of the system is good enough, then most likely different parts / layers of it are decoupled via interfaces so static analysis of the CA will not give us a true picture. I guess it’s one of the main reasons why CA is not that popular in tools like Sonar. Fortunately, it’s totally fine for us since, if you remember, we are interested in applying this specifically to old ugly code bases.

In general, I’d say that runtime analysis would give much better results, but unfortunately it’s much more costly, time consuming, and complex, so our approach is a potentially useful and lower cost alternative.

This article was written by Andrey Shalitkin, a Toptal Java developer.

A New Way of Using Email for Support Apps: An AWS Tutorial


Email may not be as cool as other communication platforms but working with it can still be fun. I was recently tasked with implementing messaging in a mobile app. The only catch was that the actual communication needed to be over email. We wanted app users to be able to communicate with a support team just like you would send a text message. Support team members needed to receive these messages via email, and also needed to be able to respond to the originating user. To the end user, everything needed to look and function just like any other modern messaging app.

In this article, we will take a look at how to implement a service similar to the one described above using Java and a handful of Amazon’s web services. You will need a valid AWS account, a domain name, and access to your favorite Java IDE.

The Infrastructure

Before we write any code, we’re going to set up the required AWS services for routing and consuming email. We’re going to use SES for sending and consuming emails and SNS+SQS for routing incoming messages.

Consuming Email Programmatically Using AWS

Revitalize e-mail in support applications with Amazon SES.

It all starts here with SES. Start by logging into your AWS account and navigating to the SES console.

Before we begin, you’re going to need a verified domain name you can send emails from.

This will be the domain app users will be sending email messages from and support members will be replying to. Verifying a domain with SES is a straightforward process, and more info can be found here.

If this is the first time you are using SES, or you have not requested a sending limit, your account will be sandboxed. This means that you will not be able to send email to addresses that aren’t verified with AWS. This may cause an error later in the tutorial, when we send an email to our fictional help desk. To avoid this, you can verify whatever email address you plan on using as your help desk in the SES console in the Email Addresses tab.

Once you have a verified domain, we can create a rule set. Navigate to the Rule Sets tab in the SES console and create a new Receipt Rule.

The first step when creating a receipt rule will be defining a recipient.

Recipients filters will allow you to define what emails SES will consume, and how to process each incoming message. The recipient we define here needs to match the domain and address pattern app user messages are emailed from. The simplest case here would be to add a recipient for the domain we previously verified, in our case example.com. This will configure SES to apply our rule to all emails sent to example.com. (e.g. foo@example.com, bar@example.com).

To create a rule for our entire domain, we would add a recipient for example.com.

It’s also possible to match address patterns. This is useful if you want to route incoming messages to different SQS queues.

Say that we have queue A and queue B. We could add two recipients: a@example.com and b@example.com. If we want to insert a message into queue A, we would email a+foo@example.com. The a part of this will match our a@example.com recipient. Everything between the + and @ is arbitrary user data, it will not affect SES’s address matching. To insert into queue B, simply replace a with b.

After you define your recipients, the next step is to configure the action SES will perform after consuming a new email. We eventually want these to end up in SQS, however it is currently not possible to go directly from SES to SQS. To bridge the gap, we need to use SNS. Select the SNS action and create a new topic. We will eventually configure this topic to insert messages into SQS.

Select create SNS topic and give it a name.

After the topic is created, we need to select a message encoding. I’m going to use Base64 in order to preserve special characters. The encoding you choose will affect how messages are decoded when we consume them in our service.

Once the rule is set, we just need to name it.

The next step will be configuring SQS and SNS, for that we need to head over to the SQS console and create a new queue.

To keep things simple, I’m using the same name as our SNS topic.

After we define our queue, we’re going to need to adjust its access policy. We only want to grant our SNS topic permission to insert. We can achieve this by adding a condition that matches our SNS topic arn.

The value field should be populated with the ARN for the SNS topic SES is notifying.

After SQS is set up, it’s time for one more trip back to the SNS console to configure your topic to insert notifications into your shiny new SQS queue.

In the SNS console, select the topic SES is notifying. From there, create a new subscription. The subscription protocol should be Amazon SQS, and the destination should be the ARN of the SQS queue you just generated.

After all that, the AWS side of the equation should be all set up. We can test our work by emailing ourselves. Send an email to the domain configured with SES, then head to the SQS console and select your queue. You should be able to see the payload containing your email.

Java Service to Deal with Emails

Now on to the fun part! In this section, we’re going to create a simple microservice capable of sending messages and processing incoming emails. The first step will be defining an API that will email our support desk on behalf of a user.

A quick note. We’re going to focus on the business logic components of this service, and won’t be defining REST endpoints or a persistence layer.

To build a Spring service, we’re going to use Spring Boot and Maven. We can use Spring Initializer to generate a project for us, start.spring.io.

To start, our pom.xml should look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.toptal.tutorials</groupId>
	<artifactId>email-processor</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<name>email-processor</name>
	<description>A simple "micro-service" for emailing support on behalf of a user and processing replies</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.3.5.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

	
	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

</project>

Emailing Support on Behalf of a User

First, let’s define a bean for emailing our support desk on behalf of a user. The job of this bean will be to process an incoming message from a user ID, and email that message to our pre-defined support desk email address.

Let’s start by defining an interface.

public interface SupportBean {

    /**
     * Send a message to the application support desk on behalf of a user
     * @param fromUserId The ID of the originating user
     * @param message The message to send
     */
    void messageSupport(long fromUserId, String message);
}

And an empty implementation:

@Component
public class SupportBeanSesImpl implements SupportBean {

    /**
     * Email address for our application help-desk
     * This is the destination address user support emails will be sent to
     */
    private static final String SUPPORT_EMAIL_ADDRESS = "support@example.com";

    @Override
    public void messageSupport(long fromUserId, String message) {
        //todo: send an email to our support address
    }
}

Let’s also add the AWS SDK to our pom, we’re going to use the SES client to send our emails:

<dependency>
	<groupId>com.amazonaws</groupId>
	<artifactId>aws-java-sdk</artifactId>
	<version>1.11.5</version>
</dependency>

The first thing we need to do is generate an email address to send our user’s message from. The address we generate will play a critical role on the consuming side of our service. It needs to contain enough information to route the help desk’s reply back to the originating user.

To achieve this, we’re going to include the originating user ID in our generated email address. To keep things clean, we’re going to create an object containing the user ID and use the Base64 encoded JSON string of it as the email address.

Let’s create a new bean responsible for turning a user ID into an email address.

public interface UserEmailBean {

    /**
     * Returns a unique per user email address
     * @param userID Input user ID
     * @return An email address unique for the input userID
     */
    String emailAddressForUserID(long userID);
}

Let’s start our implementation by adding the required consents and a simple inner class that will help us serialize our JSON.

@Component
public class UserEmailBeanJSONImpl implements UserEmailBean {

    /**
     * The TLD for all our generated email addresses
     */
    private static final String EMAIL_DOMAIN = "example.com";

    /**
     * com.fasterxml.jackson.databind.ObjectMapper used to create a JSON object including our user ID
     */
    private final ObjectMapper objectMapper = new ObjectMapper();

    @Override
    public String emailAddressForUserID(long userID) {
//todo: create the email address
        return null;
    }

    /**
     * Simple helper class we will serialize.
     * The JSON representation of this class will become our user email address
     */
    private static class UserDetails{
        private Long userID;

        public Long getUserID() {
            return userID;
        }

        public void setUserID(Long userID) {
            this.userID = userID;
        }
    }
}

Generating our email address is straightforward, all we need to do is create a UserDetails object and Base64 encode the JSON representation. The finished version of our createAddressForUserID method should look something like this:

   @Override
    public String emailAddressForUserID(long userID) {
        UserDetails userDetails = new UserDetails();
        userDetails.setUserID(userID);
        //create a JSON representation.
        String jsonString = objectMapper.writeValueAsString(userDetails);
        //Base64 encode it
        String base64String = Base64.getEncoder().encodeToString(jsonString.getBytes());
        //create an email address out of it
        String emailAddress = base64String + "@" + EMAIL_DOMAIN;
        return emailAddress;
    }

Now we can head back to SupportBeanSesImpl and update it to use the new email bean we just created.

private final UserEmailBean userEmailBean;

@Autowired
public SupportBeanSesImpl(UserEmailBean userEmailBean) {
        this.userEmailBean = userEmailBean;
}


@Override
public void messageSupport(long fromUserId, String message) throws JsonProcessingException {
        //user specific email
        String fromEmail = userEmailBean.emailAddressForUserID(fromUserId);
}

To send emails, we’re going to use the AWS SES client included with the AWS SDK.

  /**
     * SES client
     */
    private final AmazonSimpleEmailService amazonSimpleEmailService = new AmazonSimpleEmailServiceClient(
            new DefaultAWSCredentialsProviderChain() //see http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
    );

We’re utilizing the DefaultAWSCredentialsProviderChain to manage credentials for us, this class will search for AWS credentials as defined here.

We’re going to an AWS access key provisioned with access to SES and eventually SQS. For more info check out the documentation from Amazon.

The next step will be updating our messageSupport method to email support using the AWS SDK. The SES SDK makes this a straightforward process. The finished method should look something like this:

@Override
public void messageSupport(long fromUserId, String message) throws JsonProcessingException {
        //User specific email
        String fromEmail = userEmailBean.emailAddressForUserID(fromUserId);

        //create the email 
        Message supportMessage = new Message(
                new Content("New support request from userID " + fromUserId), //Email subject
                new Body().withText(new Content(message)) //Email body, this contains the user’s message
        );
        
        //create the send request
        SendEmailRequest supportEmailRequest = new SendEmailRequest(
                fromEmail, //From address, our user's generated email
                new Destination(Collections.singletonList(SUPPORT_EMAIL_ADDRESS)), //to address, our support email address
                supportMessage //Email body defined above
        );
        
        //Send it off
        amazonSimpleEmailService.sendEmail(supportEmailRequest);
    }

To try it out, create a test class and inject the SupportBean. Make sure SUPPORT_EMAIL_ADDRESS defined in SupportBeanSesImpl points to an email address you own. If your SES account is sandboxed, this address also needs to be verified. Email addresses can be verified in the SES console under Email Addresses section.

@Test
public void emailSupport() throws JsonProcessingException {
	supportBean.messageSupport(1, "Hello World!");
}

After running this, you should see a message show up in your inbox. Better yet, reply to the message and check the SQS queue we set up earlier. You should see a payload containing your reply.

Consuming Replies from SQS

The last step will be to read in emails from SQS, parse out the email message, and figure out what user ID the reply should be forwarded belongs to.

Message queueing services like Amazon SQS play a vital role in service-oriented architecture by allowing services to communicate with each other without having to compromise speed, reliability or scalability.

To listen for new SQS messages, we’re going to use the Spring Cloud AWS messaging SDK. This will allow us to configure a SQS message listener via annotations, and thus avoid quite a bit of boilerplate code.

First, the required dependencies.

Add the Spring Cloud messaging dependency:

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-aws-messaging</artifactId>
</dependency>

And add Spring Cloud AWS to your pom dependency management:

<dependencyManagement>
	<dependencies>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-aws</artifactId>
			<version>1.1.0.RELEASE</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
	</dependencies>
</dependencyManagement>

Currently, Spring Cloud AWS doesn’t support annotation driven configuration, so we’re going to have to define an XML bean. Luckily we don’t need much configuration at all, so our bean definition will be pretty light. The main point of this file will be to enable annotation driven queue listeners, this will allow us to annotate a method as an SqsListener.

Create a new XML file named aws-config.xml in your resources folder. Our definition should look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:aws-context="http://www.springframework.org/schema/cloud/aws/context"
       xmlns:aws-messaging="http://www.springframework.org/schema/cloud/aws/messaging"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/cloud/aws/context
       http://www.springframework.org/schema/cloud/aws/context/spring-cloud-aws-context.xsd
       http://www.springframework.org/schema/cloud/aws/messaging
     http://www.springframework.org/schema/cloud/aws/messaging/spring-cloud-aws-messaging.xsd">

    <!--enable annotation driven queue listeners -->
    <aws-messaging:annotation-driven-queue-listener />
    <!--define our region, this lets us reference queues by name instead of by URL. -->
    <aws-context:context-region region="us-east-1" />

</beans>

The important part of this file is <aws-messaging:annotation-driven-queue-listener />. We are also defining a default region. This is not necessary, but doing so will allow us to reference our SQS queue by name instead of URL. We are not defining any AWS credentials, by omitting them Spring will default to DefaultAWSCredentialsProviderChain, the same provider we used earlier in our SES bean. More info can be found in the Spring Cloud AWS docs.

To use this XML config in our Spring Boot app, we need to explicitly import it. Head over to your @SpringBootApplication class and import it.

@SpringBootApplication
@ImportResource("classpath:aws-config.xml") //Explicit import for our AWS XML bean definition
public class EmailProcessorApplication {

	public static void main(String[] args) {
		SpringApplication.run(EmailProcessorApplication.class, args);
	}
}

Now let’s define a bean that will handle incoming SQS messages. Spring Cloud AWS lets us accomplish this with a single annotation!

/**
 * Bean reasonable for polling SQS and processing new emails
 */
@Component
public class EmailSqsListener {

    @SuppressWarnings("unused") //IntelliJ isn't quite smart enough to recognize methods marked with @SqsListener yet
    @SqsListener(value = "com-example-ses", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)   //Mark this method as a SQS listener
                                                                                                    //Since we already set up our region we can use the logical queue name here
                                                                                                    //Spring will automatically delete messages if this method executes successfully
    public void consumeSqsMessage(@Headers Map<String, String> headers, //Map of headers returned when requesting a message from SQS
                                                                        //This map will include things like the relieved time, count and message ID
                                  @NotificationMessage String rawJsonMessage   //JSON string representation of our payload
                                                                            //Spring Cloud AWS supports marshaling here as well
                                                                            //For the sake of simplicity we will work with incoming messages as a JSON object
    ) throws Exception{

        //com.amazonaws.util.json.JSONObject included with the AWS SDK
        JSONObject jsonSqsMessage = new JSONObject(rawJsonMessage);

    }
}

The magic here lies with the @SqsListener annotation. With this, Spring will set up an Executor and start polling SQS for us. Every time a new message is found, our annotated method will be invoked with the message contents. Optionally, Spring Cloud can be configured to marshall incoming messages, giving you the ability to work with strong typed objects inside your queue listener. Additionally, you have the ability to inject a single header or a map of all headers returned from the underlying AWS call.

We’re able to use the logical queue name here since we previously defined the region in aws-config.xml, if we wanted to omit that we would be able to replace the value with our fully qualified SQS URL. We’re also defining a deletion policy, this will configure Spring to delete the incoming message from SQS if its condition is met. There are multiple policies defined in SqsMessageDeletionPolicy, we’re configuring Spring to delete our message if our consumeSqsMessage method executes successfully.

We’re also injecting the returned SQS headers into our method using @Headers, and the injected map will contain metadata related to the queue and payload received. The message body is injected using @NotificationMessage. Spring supports marshalling utilizing Jackson, or via a custom message body converter. For the sake of convenience, we’re just going to inject the raw JSON string and work with it using the JSONObject class included with the AWS SDK.

The payload retrieved from SQS will contain a lot of data. Take a look at the JSONObject to familiarize yourself with the payload returned. Our payload contains data from every AWS service it was passed through, SES, SNS, and finally SQS. For the sake of this tutorial, we really only care about two things: the list of email addresses this was sent to and the email body. Let’s start by parsing out the emails.

//Pull out the array containing all email addresses this was sent to
JSONArray emailAddressArray = jsonSqsMessage.getJSONObject("mail").getJSONArray("destination");
for(int i = 0 ; i < emailAddressArray.length() ; i++){
	String emailAddress = emailAddressArray.getString(i);
}

Since in the real world, our helpdesk may include more than just the original sender in his or her reply, we’re going to want to verify the address before we parse out the user ID. This will give our support desk both the ability to message multiple users at the same time as well as the ability to include non app users .

Let’s head back over to our UserEmailBean interface and add another method.

/**
 * Returns true if the input email address matches our template
 * @param emailAddress Email to check
 * @return true if it matches
 */
boolean emailMatchesUserFormat(String emailAddress);

In UserEmailBeanJSONImpl, to implement this method we’re going to want to do two things. First, check if the address ends with our EMAIL_DOMAIN, then check if we can marshall it.

   @Override
    public boolean emailMatchesUserFormat(String emailAddress) {

        //not our address, return right away
        if(!emailAddress.endsWith("@" + EMAIL_DOMAIN)){
            return false;
        }
        //We just care about the email part, not the domain part
        String emailPart = splitEmail(emailAddress);
        try {
            //Attempt to decode our email
            UserDetails userDetails = objectMapper.readValue(Base64.getDecoder().decode(emailPart), UserDetails.class);
            //We assume this email matches if the address is successfully decoded and marshaled  
            return userDetails != null && userDetails.getUserID() != null;
        } catch (IllegalArgumentException | IOException e) {
            //The Base64 decoder will throw an IllegalArgumentException it the input string is not Base64 formatted
            //Jackson will throw an IOException if it can't read the string into the UserDetails class
            return false;
        }
    }
    /**
     * Splits an email address on @
     * Returns everything before the @
     * @param emailAddress Address to split
     * @return all parts before @. If no @ is found, the entire address will be returned
     */
    private static String splitEmail(String emailAddress){
        if(!emailAddress.contains("@")){
            return emailAddress;
        }
        return emailAddress.substring(0, emailAddress.indexOf("@"));
    }

We defined two new methods, emailMatchesUserFormat which we just added to our interface, and a simple utility method for splitting an email address on the @. Our emailMatchesUserFormat implementation works by attempting to Base64 decode and marshall the address part back into our UserDetails helper class. If this succeeds, we then check to make sure the required userID is populated. If all this works out, we can safely assume a match.

Head back to our EmailSqsListener and inject the freshly updated UserEmailBean.

   private final UserEmailBean userEmailBean;

    @Autowired
    public EmailSqsListener(UserEmailBean userEmailBean) {
        this.userEmailBean = userEmailBean;
    }

Now we’re going to update the consumeSqsMethod. First let’s parse out the email body:

 //Pull our content, remember the content will be Base64 encoded as per our SES settings
        String encodedContent = jsonSqsMessage.getString("content");
        //Create a new String after decoding our body
        String decodedBody = new String(
                Base64.getDecoder().decode(encodedContent.getBytes())      
        );
        for(int i = 0 ; i < emailAddressArray.length() ; i++){
            String emailAddress = emailAddressArray.getString(i);
        }

Now let’s create a new method that will process the email address and email body.

private void processEmail(String emailAddress, String emailBody){
        
}

And finally, update the email loop to invoke this method if it finds a match.

//Loop over all sent to addresses
for(int i = 0 ; i < emailAddressArray.length() ; i++){
    String emailAddress = emailAddressArray.getString(i);
    //If we find a match, process the email and method
    if(userEmailBean.emailMatchesUserFormat(emailAddress)){
        processEmail(emailAddress, decodedBody);
    }
}

Before we implement processEmail, we need to add one more method to our UserEmailBean. We need a method for returning the userID from an email. Head back over to the UserEmailBean interface to add its last method.

   /**
     * Returns the userID from a formatted email address.
     * Returns null if no userID is found. 
     * @param emailAddress Formatted email address, this address should be verified using {@link #emailMatchesUserFormat(String)}
     * @return The originating userID if found, null if not
     */
    Long userIDFromEmail(String emailAddress);

The goal of this method will be to return the userID from a formatted address. The implementation will be similar to our verification method. Let’s head over to UserEmailBeanJSONImpl and fill in this method.

   @Override
    public Long userIDFromEmail(String emailAddress) {
        String emailPart = splitEmail(emailAddress);
        try {
            //Attempt to decode our email
            UserDetails userDetails = objectMapper.readValue(Base64.getDecoder().decode(emailPart), UserDetails.class);
            if(userDetails == null || userDetails.getUserID() == null){
                //We couldn't find a userID
                return null;
            }
            //ID found, return it
            return userDetails.getUserID();
        } catch (IllegalArgumentException | IOException e) {
            //The Base64 decoder will throw an IllegalArgumentException it the input string is not Base64 formatted
            //Jackson will throw an IOException if it can't read the string into the UserDetails class
            //Return null since we didn't find a userID
            return null;
        }
    }

Now head back over to our EmailSqsListener and update processEmail to use this new method.

private void processEmail(String emailAddress, String emailBody){
    //Parse out the email address
    Long userID = userEmailBean.userIDFromEmail(emailAddress);
    if(userID == null){
        //Whoops, we couldn't find a userID. Abort!
        return;
    }
}

Great! Now we have almost everything we need. The last thing we need to do is parse out the reply from the raw message.

Email clients, just like web browsers from a few years ago, are plagued by the inconsistencies in their implementations.

Parsing out replies from emails is actually a fairly complicated task. Email message formats are not standardized, and the variations between different email clients can be huge. The raw response is also going to include much more than the reply and a signature. The original message will most likely be included as well. Smart people over at Mailgun put together a great blog post explaining some of the challenges. They also open sourced their machine-learning based approach to parsing emails, check it out here.

The Mailgun library is written in Python, so for our tutorial we’re going to use a simpler Java based solution. GitHub user edlio put together an MIT licensed email parser in Java based on one of GitHub’s libraries. We’re going to use this great library.

First let’s update our pom, we’re going to use https://jitpack.io to pull in EmailReplyParser.

<repositories>
	<repository>
		<id>jitpack.io</id>
		<url>https://jitpack.io</url>
	</repository>
</repositories>

Now add the GitHub dependency.

<dependency>
	<groupId>com.github.edlio</groupId>
	<artifactId>EmailReplyParser</artifactId>
	<version>v1.0</version>
</dependency>

We’re also going to use Apache commons email. We’re going to need to parse the raw email into a javax.mail MimeMessage before passing it off to the EmailReplyParser. Add the commons dependency.

<dependency>
	<groupId>org.apache.commons</groupId>
	<artifactId>commons-email</artifactId>
	<version>1.4</version>
</dependency>

Now we can head back over to our EmailSqsListener and finish up processEmail. At this point, we have the originating userID and the raw email body. The only thing left to do is parse out the reply.

To accomplish this, we’re going to use a combination of javax.mail and edlio’s EmailReplyParser.

private void processEmail(String emailAddress, String emailBody) throws Exception {
        //Parse out the email address
        Long userID = userEmailBean.userIDFromEmail(emailAddress);
        if(userID == null){
            //Whoops, we couldn't find a userID. Abort!
            return;
        }

        //Default javax.mail session
        Session session = Session.getDefaultInstance(new Properties());
        //Create a new mimeMessage out of the raw email body
        MimeMessage mimeMessage = MimeMessageUtils.createMimeMessage(
                session,
                emailBody
        );
        MimeMessageParser mimeMessageParser = new MimeMessageParser(mimeMessage);
        //Parse the message
        mimeMessageParser.parse();
        //Parse out the reply for our message
        String replyText = EmailReplyParser.parseReply(mimeMessageParser.getPlainContent());
        //Now we're done!
        //We have both the userID and the response!
        System.out.println("Processed reply for userID: " + userID + ". Reply: " + replyText);
    }

Wrap Up

And that’s it! We now have everything we need to deliver a response to the originating user!

See? I told you email can be fun!

In this article, we saw how Amazon Web Services can be used to orchestrate complex pipelines. Although in this article, the pipeline was designed around emails; these same tools can be leveraged to design even more complex systems, where you don’t have to worry about maintaining the infrastructure and can focus on the fun aspects of software engineering instead.

This article was written by Francis Altomare, a Toptal Java developer.