Handling Data Requests in Open Event Organizer Android App

Open Event Organizer is a client side application of Open Event API Server created for event organizers and entry managers. The app maintains a local database and syncs it with the server when required. I will be talking about handling data requests in the app in this blog.

The app uses ReactiveX for all the background tasks including data accessing. When a user requests any data, there are two possible ways the app can perform. The one where app fetches the data directly from the local database maintained and another where it requests data from the server. The app has to decide one of the ways. In the Organizer app, AbstractObservableBuilder class takes care of this. The relevant code is:

final class AbstractObservableBuilder<T> {

   private final IUtilModel utilModel;
   private boolean reload;
   private Observable<T> diskObservable;
   private Observable<T> networkObservable;

   ...
   ...

   @NonNull
   private Callable<Observable<T>> getReloadCallable() {
       return () -> {
           if (reload)
               return Observable.empty();
           else
               return diskObservable
                   .doOnNext(item -> Timber.d("Loaded %s From Disk on Thread %s",
                       item.getClass(), Thread.currentThread().getName()));
       };
   }

   @NonNull
   private Observable<T> getConnectionObservable() {
       if (utilModel.isConnected())
           return networkObservable
               .doOnNext(item -> Timber.d("Loaded %s From Network on Thread %s",
                   item.getClass(), Thread.currentThread().getName()));
       else
           return Observable.error(new Throwable(Constants.NO_NETWORK));
   }

   @NonNull
   private <V> ObservableTransformer<V, V> applySchedulers() {
       return observable -> observable
           .subscribeOn(Schedulers.io())
           .observeOn(AndroidSchedulers.mainThread());
   }

   @NonNull
   public Observable<T> build() {
       if (diskObservable == null || networkObservable == null)
           throw new IllegalStateException("Network or Disk observable not provided");

       return Observable
               .defer(getReloadCallable())
               .switchIfEmpty(getConnectionObservable())
               .toList()
               .flatMap(items -> diskObservable.toList())
               .flattenAsObservable(items -> items)
               .compose(applySchedulers());
   }
}

 

DiskObservable is a data request to the local database and networkObservable is a data request to the server. The build function decides which one to use and returns a correct observable accordingly. The class object takes a boolean field reload which is used to decide which observable to subscribe. If reload is true, that means the user wants data from the server, hence networkObservable is returned to subscribe. Also switchIfEmpty in the build method checks whether the data fetched using diskObservable is empty, if found empty it switches the observable to the networkObservable to subscribe.

This class object is used for every data access in the app. For example, this is a code snippet of the gettEvents method in EventRepository class.

@Override
public Observable<Event> getEvents(boolean reload) {
   Observable<Event> diskObservable = Observable.defer(() ->
       databaseRepository.getAllItems(Event.class)
   );

   Observable<Event> networkObservable = Observable.defer(() ->
       eventService.getEvents(JWTUtils.getIdentity(getAuthorization()))
           ...
           ...
           .flatMapIterable(events -> events));

   return new AbstractObservableBuilder<Event>(utilModel)
       .reload(reload)
       .withDiskObservable(diskObservable)
       .withNetworkObservable(networkObservable)
       .build();
}

 

Links:
1. Documentation of ReactiveX API
2. Github repository link of RxJava – Reactive Extension for JVM

Keeping Order of tickets in Event Wizard in Sync with API on Open Event Frontend

This blog article will illustrate how the various tickets are stored and displayed in order the event organiser decides  on  Open Event Frontend and also, how they are kept in sync with the backend.

First we will take a look at how the user is able to control the order of the tickets using the ticket widget.

{{#each tickets as |ticket index|}}
  {{widgets/forms/ticket-input ticket=ticket
  timezone=data.event.timezone
  canMoveUp=(not-eq index 0)
  canMoveDown=(not-eq ticket.position (dec
  data.event.tickets.length))
  moveTicketUp=(action 'moveTicket' ticket 'up')
  moveTicketDown=(action 'moveTicket' ticket 'down')
  removeTicket=(confirm 'Are you sure you  wish to delete this 
  ticket ?' (action 'removeTicket' ticket))}}
{{/each}}

The canMoveUp and canMoveDown are dynamic properties and are dependent upon the current positions of the tickets in the tickets array.  These properties define whether the up or down arraow or both should be visible alongside the ticket to trigger the moveTicket action.

There is an attribute called position in the ticket model which is responsible for storing the position of the ticket on the backend. Hence it is necessary that the list of the ticket available should always be ordered by position. However, it should be kept in mind, that even if the position attribute of the tickers is changed, it will not actually change the indices of the ticket records in the array fetched from the API. And since we want the ticker order in sync with the backend, i.e. user shouldn’t have to refresh to see the changes in ticket order, we are going to return the tickets via a computed function which sorts them in the required order.

tickets: computed('data.event.tickets.@each.isDeleted', 'data.event.tickets.@each.position', function() {
   return this.get('data.event.tickets').sortBy('position').filterBy('isDeleted', false);
 })

The sortBy method ensures that the tickets are always ordered and this computed property thus watches the position of each of the tickets to look out for any changes. Now we can finally define the moveTicket action to enable modification of position for tickets.

moveTicket(ticket, direction) {
     const index = ticket.get('position');
     const otherTicket = this.get('data.event.tickets').find(otherTicket => otherTicket.get('position') === (direction === 'up' ? (index - 1) : (index + 1)));
     otherTicket.set('position', index);
     ticket.set('position', direction === 'up' ? (index - 1) : (index + 1));
   }

The moveTicket action takes two arguments, ticket and direction. It temporarily stores the position of the current ticket and the position of the ticket which needs to be swapped with the current ticket.Based on the direction the positions are swapped. Since the position of each of the tickets is being watched by the tickets computed array, the change in order becomes apparent immediately.

Now when the User will trigger the save request, the positions of each of the tickets will be updated via a PATCH or POST (if the ticket is new) request.

Also, the positions of all the tickets maybe affected while adding a new ticket or deleting an existing one. In case of a new ticket, the position of the new ticket should be initialised while creating it and it should be below all the other tickets.

addTicket(type, position) {
     const salesStartDateTime = moment();
     const salesEndDateTime = this.get('data.event.startsAt');
     this.get('data.event.tickets').pushObject(this.store.createRecord('ticket', {
       type,
       position,
       salesStartsAt : salesStartDateTime,
       salesEndsAt   : salesEndDateTime
     }));
   }

Deleting a ticket requires updating positions of all the tickets below the deleted ticket. All of the positions need to be shifted one place up.

removeTicket(deleteTicket) {
     const index = deleteTicket.get('position');
     this.get('data.event.tickets').forEach(ticket => {
       if (ticket.get('position') > index) {
         ticket.set('position', ticket.get('position') - 1);
       }
     });
     deleteTicket.deleteRecord();
   }

The tickets whose position is to be updated are filtered by comparison of their position from the position of the deleted ticket.

Resources

Implementing Order Statistics API on Tickets Route in Open Event Frontend

The order statistics API endpoints are used to display the statistics related to tickets, orders, and sales. It contains the details about the total number of orders, the total number of tickets sold and the amount of the sales. It also gives the detailed information about the pending, expired, placed and completed orders, tickets, and sales.

This article will illustrate how the order statistics can be displayed using the Order Statistics API in Open Event Frontend. The primary end point of Open Event API with which we are concerned with for statistics is

GET /v1/events/{event_identifier}/order-statistics

First, we need to create a model for the order statistics, which will have the fields corresponding to the API, so we proceed with the ember CLI command:

ember g model order-statistics-tickets

Next, we need to define the model according to the requirements. The model needs to extend the base model class. The code for the model looks like this:

import attr from 'ember-data/attr';
import ModelBase from 'open-event-frontend/models/base';

export default ModelBase.extend({
  orders  : attr(),
  tickets : attr(),
  sales   : attr()
});

As we need to display the statistics related to orders, tickets, and sales so we have their respective variables inside the model which will fetch and store the details from the API.

Now, after creating a model, we need to make an API call to get the details. This can be done using the following:

return this.modelFor('events.view').query('orderStatistics', {});

Since the tickets route is nested inside the event.view route so, first we are getting the model for event.view route and then we’re querying order statistics from the model.

The complete code can be seen here.

Now, we need to call the model inside the template file to display the details. To fetch the total orders we can write like this

{{model.orders.total}}

 

In a similar way, the total sales can be displayed like this.

{{model.sales.total}}

 

And total tickets can be displayed like this

{{model.tickets.total}}

 

If we want to fetch other details like the pending sales or completed orders then the only thing we need to replace is the total attribute. In place of total, we can add any other attribute depending on the requirement. The complete code of the template can be seen here.

The UI for the order statistics on the tickets route looks like this.

Fig. 1: The user interface for displaying the statistics

The complete source code can be seen here.

Resources:

Implementing Pages API in Open Event Frontend

The pages endpoints are used to create static pages which such as about page or any other page that doesn’t need to be updated frequently and only a specific content is to be shown. This article will illustrate how the pages can be added or removed from the /admin/content/pages route using the pages API in Open Event Frontend. The primary end point of Open Event API with which we are concerned with for pages is

GET /v1/pages

First, we need to create a model for the pages, which will have the fields corresponding to the API, so we proceed with the ember CLI command:

ember g model page

Next, we need to define the model according to the requirements. The model needs to extend the base model class. The code for the page model looks like this:

import attr from 'ember-data/attr';
import ModelBase from 'open-event-frontend/models/base';

export default ModelBase.extend({
  name        : attr('string'),
  title       : attr('string'),
  url         : attr('string'),
  description : attr('string'),
  language    : attr('string'),
  index       : attr('number', { defaultValue: 0 }),
  place       : attr('string')
});

As the page will have name, title, url which will tell the URL of the page, the language, the description, index and the place of the page where it has to be which can be either a footer or an event.

The complete code for the model can be seen here.

Now, after creating a model, we need to make an API call to get and post the pages created. This can be done using the following:

return this.get('store').findAll('page');

The above line will check the store and find all the pages which have been cached in and if there is no record found then it will make an API call and cache the records in the store so that when called it can return it immediately.

Since in the case of pages we have multiple options like creating a new page, updating a new page, deleting an existing page etc. For creating and updating the page we have a form which has the fields required by the API to create the page.  The UI of the form looks like this.

Fig. 1: The user interface of the form used to create the page.

Fig. 2: The user interface of the form used to update and delete the already existing page

The code for the above form can be seen here.

Now, if we click the items which are present in the sidebar on the left, it enables us to edit and update the page by displaying the information stored in the form and then the details be later updated on the server by clicking the Update button. If we want to delete the form we can do so using the delete button which first shows a pop up to confirm whether we actually want to delete it or not. The code for displaying the delete confirmation pop up looks like this.

<button class="ui red button" 
{{action (confirm (t 'Are you sure you would like to delete this page?') (action 'deletePage' data))}}>
{{t 'Delete'}}</button>

 

The code to delete the page looks like this

deletePage(data) {
    if (!this.get('isCreate')) {
      data.destroyRecord();
      this.set('isFormOpen', false);
    }
  }

In the above piece of code, we’re checking whether the form is in create mode or update mode and if it’s in create mode then we can destroy the record and then close the form.

The UI for the pop up looks like this.

Fig.3: The user interface for delete confirmation pop up

The code for the entire process of page creation to deletion can be checked here

To conclude, this is how we efficiently do the process of page creation, updating and deletion using the Open-Event-Orga pages API  ensuring that there is no unnecessary API call to fetch the data and no code duplication.

Resources:

Using Android Palette with Glide in Open Event Organizer Android App

Open Event Organizer is an Android Application for the Event Organizers and Entry Managers. The core feature of the App is to scan a QR code from the ticket to validate an attendee’s check in. Other features of the App are to display an overview of sales, ticket management and basic editing in the Event Details. Open Event API Server acts as a backend for this App. The App uses Navigation Drawer for navigation in the App. The side drawer contains menus, event name, event start date and event image in the header. Event name and date is shown just below the event image in a palette. For a better visibility Android Palette is used which extracts prominent colors from images. The App uses Glide to handle image loading hence GlidePalette library is used for palette generation which integrates Android Palette with Glide. I will be talking about the implementation of GlidePalette in the App in this blog.

The App uses Data Binding so the image URLs are directly passed to the XML views in the layouts and the image loading logic is implemented in the BindingAdapter class. The image loading code looks like:

GlideApp
   .with(imageView.getContext())
   .load(Uri.parse(url))
   ...
   .into(imageView);

 

So as to implement palette generation for event detail label, it has to be implemented with the event image loading. GlideApp takes request listener which implements methods on success and failure where palette can be generated using the bitmap loaded. With GlidePalette most of this part is covered in the library itself. It provides GlidePalette class which is a sub class of GlideApp request listener which is passed to the GlideApp using the method listener. In the App, BindingAdapter has a method named bindImageWithPalette which takes a view container, image url, a placeholder drawable and the ids of imageview and palette. The relevant code is:

@BindingAdapter(value = {"paletteImageUrl", "placeholder", "imageId", "paletteId"}, requireAll = false)
public static void bindImageWithPalette(View container, String url, Drawable drawable, int imageId, int paletteId) {
   ImageView imageView = (ImageView) container.findViewById(imageId);
   ViewGroup palette = (ViewGroup) container.findViewById(paletteId);

   if (TextUtils.isEmpty(url)) {
       if (drawable != null)
           imageView.setImageDrawable(drawable);
       palette.setBackgroundColor(container.getResources().getColor(R.color.grey_600));
       for (int i = 0; i < palette.getChildCount(); i++) {
           View child = palette.getChildAt(i);
           if (child instanceof TextView)
               ((TextView) child).setTextColor(Color.WHITE);
       }
       return;
   }
   GlidePalette<Drawable> glidePalette = GlidePalette.with(url)
       .use(GlidePalette.Profile.MUTED)
       .intoBackground(palette)
       .crossfade(true);

   for (int i = 0; i < palette.getChildCount(); i++) {
       View child = palette.getChildAt(i);
       if (child instanceof TextView)
           glidePalette
               .intoTextColor((TextView) child, GlidePalette.Swatch.TITLE_TEXT_COLOR);
   }
   setGlideImage(imageView, url, drawable, null, glidePalette);
}

 

The code is pretty obvious. The method checks passed URL for nullability. If null, it sets the placeholder drawable to the image view and default colors to the text views and the palette. The GlidePalette object is generated using the initializer method with which takes the image URL. The request is passed to the method setGlideImage which loads the image and passes the GlidePalette to the GlideApp as a listener. Accordingly, the palette is generated and the colors are set to the label and text views accordingly. The container view in the XML layout looks like:

<LinearLayout
   android:layout_width="match_parent"
   android:layout_height="wrap_content"
   android:orientation="vertical"
   app:paletteImageUrl="@{ event.largeImageUrl }"
   app:placeholder="@{ @drawable/header }"
   app:imageId="@{ R.id.image }"
   app:paletteId="@{ R.id.eventDetailPalette }">

 

Links:
1. Documentation for Glide Image Loading Library
2. GlidePalette Github Repository
3. Android Palette Official Documentation

Making App Name Configurable for Open Event Organizer App

Open Event Organizer is a client side android application of Open Event API server created for event organizers and entry managers. The application provides a way to configure the app name via environment variable app_name. This allows the user to change the app name just by setting the environment variable app_name to the new name. I will be talking about its implementation in the application in this blog.

Generally, in an android application, the app name is stored as a static string resource and set in the manifest file by referencing to it. In the Organizer application, the app name variable is defined in the app’s gradle file. It is assigned to the value of environment variable app_name and the default value is assigned if the variable is null. The relevant code in the manifest file is:

def app_name = System.getenv('app_name') ?: "eventyay organizer"

app/build.gradle

The default value of app_name is kept, eventyay organizer. This is the app name when the user doesn’t set environment variable app_name. To reference the variable from the gradle file into the manifest, manifestPlaceholders are defined in the gradle’s defaultConfig. It is a map of key value pairs. The relevant code is:

defaultConfig {
   ...
   ...
   manifestPlaceholders = [appName: app_name]
}

app/build.gradle

This makes appName available for use in the app manifest. In the manifest file, app name is assigned to the appName set in the gradle.

<application
   ...
   ...
   android:label="${appName}"

app/src/main/AndroidManifest.xml

By this, the application name is made configurable from the environment variable.

Links:
1. ManifestPlaceholders documentation
2. Stackoverflow answer about getting environment variable in gradle

Implementing Notifications in Open Event Server

In FOSSASIA’s Open Event Server project, along with emails, almost all actions have necessary user notifications as well. So, when a new session is created or a session is accepted by the event organisers, along with the email, a user notification is also sent. Though showing the user notification is mainly implemented in the frontend site but the content to be shown and on which action to show is strictly decided by the server project.

A notification essentially helps an user to get the necessary information while staying in the platform itself and not needing to go to check his/her email for every action he performs. So unlike email which acts as a backup for the informations, notification is more of an instant thing.

The API

The Notifications API is mostly like all other JSON API endpoints in the open event project. However in Notifications API we do not allow any to send a POST request. The admin of the server is able to send a GET a request to view all the notifications that are there in the system while a user can only view his/her notification. As of PATCH we allow only the user to edit his/her notification to mark it as read or not read. Following is the schema for the API:

class NotificationSchema(Schema):
    """
    API Schema for Notification Model
    """

    class Meta:
        """
        Meta class for Notification API schema
        """
        type_ = 'notification'
        self_view = 'v1.notification_detail'
        self_view_kwargs = {'id': '<id>'}
        self_view_many = 'v1.microlocation_list_post'
        inflect = dasherize

    id = fields.Str(dump_only=True)
    title = fields.Str(allow_none=True, dump_only=True)
    message = fields.Str(allow_none=True, dump_only=True)
    received_at = fields.DateTime(dump_only=True)
    accept = fields.Str(allow_none=True, dump_only=True)
    is_read = fields.Boolean()
    user = Relationship(attribute='user',
                        self_view='v1.notification_user',
                        self_view_kwargs={'id': '<id>'},
                        related_view='v1.user_detail',
                        related_view_kwargs={'notification_id': '<id>'},
                        schema='UserSchema',
                        type_='user'
                        )


The main things that are shown in the notification from the frontend are the
title and message. The title is the text that is shown without expanding the entire notification that gives an overview about the message in case you don’t want to read the entire message. The message however provides the entire detail that is associated with the action performed. The user relationship stores which user the particular notification is related with. It is a one-to-one relationship where one notification can only belong to one user. However one user can have multiple notifications. Another important attribute is the is_read attribute. This is the only attribute that is allowed to be changed. By default, when we make an entry in the database, is_read is set to FALSE. Once an user has read the notification, a request is sent from the frontend to change is_read to TRUE.

The different actions for which we send notification are stored in the models/notification.py file as global variables.

USER_CHANGE_EMAIL = "User email"'
NEW_SESSION = 'New Session Proposal'
PASSWORD_CHANGE = 'Change Password'
EVENT_ROLE = 'Event Role Invitation'
TICKET_PURCHASED = 'Ticket(s) Purchased'
TICKET_PURCHASED_ATTENDEE = 'Ticket(s) purchased to Attendee    '
EVENT_EXPORTED = 'Event Exported'
EVENT_EXPORT_FAIL = 'Event Export Failed'
EVENT_IMPORTED = 'Event Imported'

HTML Templates

The notification title and message that is stored in the database and later served via the Notification API is created using some string formatting HTML templates. We firstly import all the global variables that represent the various actions from the notification model. Then we declare a global dict type variable named NOTIFS which stores all title and messages to be stored in the notification table.

NEW_SESSION: {
        'title': u'New session proposal for {event_name}',
        'message': u"""The event <strong>{event_name}</strong> has received
             a new session proposal.<br><br>
            <a href='{link}' class='btn btn-info btn-sm'>View Session</a>""",
        'recipient': 'Organizer',
    },


This is an example of the contents stored inside the dict. For every action, there is a dict with attributes
title, message and recipient. Title contains the brief overview of the entire notification whereas message contains a more vivid description with proper links. Recipient contains the one who receives the notification. So for example in the above code snippet, it is a notification for a new session created. The notification goes to the organizer. The title and message contains named placeholders which are later replaced by particular values using python’s .format() function.

Notification Helper

Notification helper module contains two main parts :-

  1. A parent function which saves the notification to the table related to the user to whom the notification belongs.
  2. Individual notification helper functions that are used by the APIs to save notification after various actions are performed.

Parent Function

def send_notification(user, action, title, message):
    if not current_app.config['TESTING']:
        notification = Notification(user_id=user.id,
                                    title=title,
                                    message=message,
                                    action=action
                                    )
        save_to_db(notification, msg="Notification saved")
        record_activity('notification_event', user=user, action=action, title=title)


send_notification
() is the parent function which takes as parameters user, action, title and message and stores them in the notification table in the database. The user is the one to whom the notification belongs to, action represents the particular action in an API which triggered the notification. Title and message are the contents that are shown in the frontend in the form of a notification. The frontend can implement it as a dropdown notification like facebook or a desktop notification like gitter or whatsapp. After the notification is saved we also update the activity table with the action that a notification has been saved for a user with the following action and title. Later, the Notification API mentioned in the very beginning of the blog uses this data that is being stored now and serves it as a JSON response.

Individual Functions

Apart from this, we have individual functions that uses the parent function to store notifications particular to a particular actions. For example, we have a send_notif_new_session_organizer() function which is used to save notification for all the organizers of an event that a new session has been added to their particular event. This function is called when a POST request is made in the Sessions API and the data is saved successfully. The function is executed for all the organizers of the event for which the session has been created.

def send_notif_new_session_organizer(user, event_name, link):
    message_settings = MessageSettings.query.filter_by(action=NEW_SESSION).first()
    if not message_settings or message_settings.notification_status == 1:
        notif = NOTIFS[NEW_SESSION]
        action = NEW_SESSION
        title = notif['title'].format(event_name=event_name)
        message = notif['message'].format(event_name=event_name, link=link)

        send_notification(user, action, title, message)


In the above function, we take in 3 parameters, user, event_name and link. The value of the user parameter is used to link the notification to that particular user. Event_name and link are used in the title and message of the notification that is saved in the database. Firstly in the function we check if there is certain message setting which tells that the user doesn’t want to receive notifications related to new sessions being created. If not, we proceed. We get the title and message strings from the NOTIFS dict from the system_notification.py file.

After that, using string formatting we get the actual message. For example,

u'New session proposal for {event_name}'.format(‘FOSSASIA’)

would give us a resulting string of the form:

u'New session proposal for FOSSASIA'

After this, we use this variables and send them as parameters to the send_notification() parent function to save the notification properly.

 

Reference:

Implement Email in Open Event Server

In FOSSASIA’s Open Event Server project, we send out emails when various different actions are performed using the API. For example, when a new user is created, he/she receives an email welcoming him to the server as well as an email verification email. Users get role invites from event organisers in the form of emails, when someone buys a ticket he/she gets a PDF link to the ticket as email. So as you can understand all the important informations that are necessary to be notified to the user are sent as an email to the user and sometimes to the organizer as well.

In FOSSASIA, we use sendgrid’s API or an SMTP server depending on the admin settings for sending emails. You can read more about how we use sendgrid’s API to send emails in FOSSASIA here. Now let’s dive into the modules that we have for sending the emails. The three main parts in the entire email sending are:

  1. Model – Storing the Various Actions
  2. Templates – Storing the HTML templates for the emails
  3. Email Functions – Individual functions for various different actions

Let’s go through each of these modules one by one.

Model

USER_REGISTER = 'User Registration'
USER_CONFIRM = 'User Confirmation'
USER_CHANGE_EMAIL = "User email"
INVITE_PAPERS = 'Invitation For Papers'
NEXT_EVENT = 'Next Event'
NEW_SESSION = 'New Session Proposal'
PASSWORD_RESET = 'Reset Password'
PASSWORD_CHANGE = 'Change Password'
EVENT_ROLE = 'Event Role Invitation'
SESSION_ACCEPT_REJECT = 'Session Accept or Reject'
SESSION_SCHEDULE = 'Session Schedule Change'
EVENT_PUBLISH = 'Event Published'
AFTER_EVENT = 'After Event'
USER_REGISTER_WITH_PASSWORD = 'User Registration during Payment'
TICKET_PURCHASED = 'Ticket(s) Purchased'


In the Model file, named as
mail.py, we firstly declare the various different actions for which we send the emails out. These actions are globally used as the keys in the other modules of the email sending service. Here, we define global variables with the name of the action as strings in them. These are all constant variables, which means that there value remains throughout and never changes. For example, USER_REGISTER has the value ‘User Registration’, which essentially means that anything related to the USER_REGISTER key is executed when the User Registration action occurs. Or in other words, whenever an user registers into the system by signing up or creating a new user through the API, he/she receives the corresponding emails.
Apart from this, we have the model class which defines a table in the database. We use this model class to store the actions performed while sending emails in the database. So we store the action, the time at which the email was sent, the recipient and the sender. That way we have a record about all the emails that were sent out via our server.

class Mail(db.Model):
    __tablename__ = 'mails'
    id = db.Column(db.Integer, primary_key=True)
    recipient = db.Column(db.String)
    time = db.Column(db.DateTime(timezone=True))
    action = db.Column(db.String)
    subject = db.Column(db.String)
    message = db.Column(db.String)

    def __init__(self, recipient=None, time=None, action=None, subject=None,
                 message=None):
        self.recipient = recipient
        self.time = time
        if self.time is None:
            self.time = datetime.now(pytz.utc)
        self.action = action
        self.subject = subject
        self.message = message

    def __repr__(self):
        return '<Mail %r to %r>' % (self.id, self.recipient)

    def __str__(self):
        return unicode(self).encode('utf-8')

    def __unicode__(self):
        return 'Mail %r by %r' % (self.id, self.recipient,)


The table name in which all the information is stored is named as mails. It stores the recipient, the time at which the email is sent (timezone aware), the action which initiated the email sending, the subject of the email and the entire html body of the email. In case a datetime value is sent, we use that, else we use the current time in the time field.

HTML Templates

We store the html templates in the form of key value pairs in a file called system_mails.py inside the helpers module of the API. Inside the system_mails, we have a global dict variable named MAILS as shown below.

MAILS = {
    EVENT_PUBLISH: {
        'recipient': 'Organizer, Speaker',
        'subject': u'{event_name} is Live',
        'message': (
            u"Hi {email}<br/>" +
            u"Event, {event_name}, is up and running and ready for action. Go ahead and check it out." +
            u"<br/> Visit this link to view it: {link}"
        )
    },
    INVITE_PAPERS: {
        'recipient': 'Speaker',
        'subject': u'Invitation to Submit Papers for {event_name}',
        'message': (
            u"Hi {email}<br/>" +
            u"You are invited to submit papers for event: {event_name}" +
            u"<br/> Visit this link to fill up details: {link}"
        )
    },
    SESSION_ACCEPT_REJECT: {
        'recipient': 'Speaker',
        'subject': u'Session {session_name} has been {acceptance}',
        'message': (
            u"Hi {email},<br/>" +
            u"The session <strong>{session_name}</strong> has been <strong>{acceptance}</strong> by the organizer. " +
            u"<br/> Visit this link to view the session: {link}"
        )
    },
    SESSION_SCHEDULE: {
        'recipient': 'Organizer, Speaker',
        'subject': u'Schedule for Session {session_name} has been changed',
        'message': (
            u"Hi {email},<br/>" +
            u"The schedule for session <strong>{session_name}</strong> has been changed. " +
            u"<br/> Visit this link to view the session: {link}"
        )
    },


Inside the MAILS dict, we have key-value pairs, where in keys we use the global variables from the Model to define the action related to the email template. In the value, we again have 3 different key-value pairs – recipient, subject and message. The recipient defines the group who should receive this email, the subject goes into the subject part of the email while message forms the body for the email. For subject and message we use unicode strings with named placeholders that are used later for formatting using python’s
.format() function.

Email Functions

This is the most important part of the entire email sending system since this is the place where the entire email sending functionality is implemented using the above two modules. We have all these functions inside a single file namely mail.py inside the helpers module of the API. Firstly, we import two things in this file – The global dict variable MAILS defined in the template file above, and the various global action variables defined in the model. There is one main module which is used by every other individual modules for sending the emails defined as send_email(to, action, subject, html). This function takes as parameters the email to which the email is to be sent, the subject string, the html body string along with the action to store it in the database.

Firstly we ensure that the email address for the recipient is present and isn’t an empty string. After we have ensured this, we retrieve the email service as set in the admin settings. It can either be “smtp” or “sendgrid”. The email address for the sender has different formatting depending on the email service we are using. While sendgrid uses just the email say for example “medomag20@gmail.com”, smtp uses a format  a little different like this: Medozonuo Suohu<medomag20@gmail.com>. So we set that as well in the email_from variable.

def send_email(to, action, subject, html):
    """
    Sends email and records it in DB
    """
    if not string_empty(to):
        email_service = get_settings()['email_service']
        email_from_name = get_settings()['email_from_name']
        if email_service == 'smtp':
            email_from = email_from_name + '<' + get_settings()['email_from'] + '>'
        else:
            email_from = get_settings()['email_from']
        payload = {
            'to': to,
            'from': email_from,
            'subject': subject,
            'html': html
        }

        if not current_app.config['TESTING']:
            if email_service == 'smtp':
                smtp_encryption = get_settings()['smtp_encryption']
                if smtp_encryption == 'tls':
                    smtp_encryption = 'required'
                elif smtp_encryption == 'ssl':
                    smtp_encryption = 'ssl'
                elif smtp_encryption == 'tls_optional':
                    smtp_encryption = 'optional'
                else:
                    smtp_encryption = 'none'

                config = {
                    'host': get_settings()['smtp_host'],
                    'username': get_settings()['smtp_username'],
                    'password': get_settings()['smtp_password'],
                    'encryption': smtp_encryption,
                    'port': get_settings()['smtp_port'],
                }

                from tasks import send_mail_via_smtp_task
                send_mail_via_smtp_task.delay(config, payload)


After this we create the payload containing the email address for the recipient, the email address of the sender, the subject of the email and the html body of the email.
For unittesting and any other testing we avoid email sending since that is really not required in the flow. So we check that the current app is not configured to run in a testing environment. After that we have two different implementation depending on the email service used.

SMTP

There are 3 kind of possible encryptions for the email that can be used with smtp server – tls, ssl and optional. We determine this based on the admin settings again. Also, from the admin settings we collect the host, username, password and port for the smtp server.

After this we start a celery task for sending the email. Since email sending to a number of clients can be time consuming so we do it using the celery queueing service without disturbing the main workflow of the entire system.

@celery.task(name='send.email.post.smtp')
def send_mail_via_smtp_task(config, payload):
    mailer_config = {
        'transport': {
            'use': 'smtp',
            'host': config['host'],
            'username': config['username'],
            'password': config['password'],
            'tls': config['encryption'],
            'port': config['port']
        }
    }

    mailer = Mailer(mailer_config)
    mailer.start()
    message = Message(author=payload['from'], to=payload['to'])
    message.subject = payload['subject']
    message.plain = strip_tags(payload['html'])
    message.rich = payload['html']
    mailer.send(message)
    mailer.stop()

Inside the celery task, we use the Mailer and Message classes from the marrow module of python. We configure the Mailer according to the various settings received from the admin and then use the payload to send the email.

Sendgrid

For sending email using the sendgrid API, we need to set the Bearer key which is used for authenticating the email service. This key is also defined in the admin settings. After we have set the Bearer key as the authorization header, we again initiate the celery task corresponding to the sendgrid email sending service.

@celery.task(name='send.email.post')
def send_email_task(payload, headers):
    requests.post(
        "https://api.sendgrid.com/api/mail.send.json",
        data=payload,
        headers=headers
    )


For sending the email service, all we need to do is make a POST request to the api endpoint “
https://api.sendgrid.com/api/mail.send.json” with the headers which contains the Bearer Key and the data which contains the payload containing all the information related to the recipient, sender, subject of email and the body of the email.

Apart from these, this module implements all the individual functions that are called based on the various functions that occur. For example, let’s look into the email sending function in case a new session is created.

def send_email_new_session(email, event_name, link):
    """email for new session"""
    send_email(
        to=email,
        action=NEW_SESSION,
        subject=MAILS[NEW_SESSION]['subject'].format(
            event_name=event_name
        ),
        html=MAILS[NEW_SESSION]['message'].format(
            email=email,
            event_name=event_name,
            link=link
        )
    )


This function is called inside the Sessions API, for every speaker of the session as well as for every organizer of the event to which the session is submitted. Inside this function, we use the
send_email().  But firstly we need to create the subject of the email and the message body of the email using the templates and by replacing placeholders by actual value using python formatting. MAILS[NEW_SESSION] returns a unicode string: u’New session proposal for {event_name}’ . So what we do is use the .format() function to replace {event_name} by the actual event_name received as parameter. So it is equivalent to doing something like:

u'New session proposal for {event_name}'.format(‘FOSSASIA’)

which would give us a resulting string of the form:

u'New session proposal for FOSSASIA'

Similarly, we create the html message body using the templates and the parameters received. After this is done, we make a function call to send_email()  which then sends the final email.

References:

Implementing Roles API on Open Event Frontend to Create Roles Using an External Modal

This blog article will illustrate how the roles are created via the external model  on the admin permissions page in Open Event Frontend, using the roles API. Our discussion primarily will involve the admin/permissions/index route to illustrate the process.The primary end point of Open Event API with which we are concerned with for fetching the permissions  for a user is

POST /v1/roles

First we need to create a model for the user-permissions, which will have the fields corresponding to the api, so we proceed with the ember CLI command:

ember g model role

Next we define the model according to the requirements. The model needs to extend the base model class, and has only two fields one for the title and one for the actual name of the role.

import attr from 'ember-data/attr';
import ModelBase from 'open-event-frontend/models/base';

export default ModelBase.extend({
 name           : attr('string'),
 titleName      : attr('string')
 });

Next we need to modify the existing modal to incorporate the API and creation of roles in it. It is very important to note here that using createRecord as the model will result in a major flaw. If createRecord is used and the user tries to create multiple roles, other than the first POST request all the subsequent requests will be PATCH requests and will keep on modifying the same role. To avoid this, a new record needs to be created every time the user clicks on Add Role.  We slightly modify the modal component call to pass in the name and titleName to it.

{{modals/add-system-role-modal  isOpen=isAddSystemRoleModalOpen
                                isLoading=isLoading
                                name=name
                                titleName=titleName
                                addSystemRole=(action 'addSystemRole')}}

Upon entering the details of the roles and successful validation of the form, if the user clicks the Add Role button of the modal, the action addSystemRole will be triggered. We will write the entire logic for the same in the respective controller of the route.

addSystemRole() {
     this.set('isLoading', true);
     this.get('store').createRecord('role', {
       name      : this.get('name'),
       titleName : this.get('titleName')
     }).save()
       .then(() => {
         this.set('isLoading', false);
         this.notify.success(this.l10n.t('User permissions have 
         been saved successfully.'));
         this.set('isAddSystemRoleModalOpen', false);
         this.setProperties({
           name          : null,
           roleTitleName : null
         });
       })
       .catch(()=> {
         this.set('isLoading', false);
         this.notify.error(this.l10n.t('An unexpected error has occurred.
         User permissions not saved.'));
       });
   },

At first the isLoading property is made true.This adds the semantic UI class loading to the the form,  and so the form goes in the loading state, Next, a record is created of the type role  and it’s properties are made equal to the corresponding values entered by the user.

Then save() is called, which subsequently makes a POST request to the server. If the request is successful the modal is closed by setting the isAddSystemRoleModalOpen property to false. Also, the fields of the modal are cleared for a  better user experience in case multiple roles need to be added one after the other.

In cases when  there is an error during the processing of the request the catch() block executes. And the modal is not closed. Neither are the fields cleared.

Resources

Giving Offline Support to the Open Event Organizer Android App

Open Event Organizer is an Android Application for Event Organizers and Entry Managers which uses Open Event API Server as a backend. The core feature of the App is to scan a QR code to validate an attendee’s check in. The App maintains a local database and syncs it with the server. The basic workflow of the attendee check in is – the App scans a QR code on an attendee’s ticket. The code scanned is processed to validate the attendee from the attendees database which is maintained locally. On finding, the App makes a check in status toggling request to the server. The server toggles the status of the attendee and sends back a response containing the updated attendee’s data which is updated in the local database. Everything described above goes well till the App gets a good network connection always which cannot be assumed as a network can go down sometimes at the event site. So to support the functionality even in the absence of the network, Orga App uses Job Schedulers which handle requests in absence of network and the requests are made when the network is available again. I will be talking about its implementation in the App through this blog.

The App uses the library Android-Job developed by evernote which handles jobs in the background. The library provides a class JobManager which does most of the part. The singleton of this class is initialized in the Application class. Job is the class which is where actually a background task is implemented. There can be more than one jobs in the App, hence the library requires to implement JobCreator interface which has create method which takes a string tag and the relevant Job is returned. JobCreator is passed to the JobManager in Application while initialization. The relevant code is:

JobManager.create(this).addJobCreator(new OrgaJobCreator());

Initialization of JobManager in Application class

public class OrgaJobCreator implements JobCreator {
   @Override
   public Job create(String tag) {
       switch (tag) {
           case AttendeeCheckInJob.TAG:
               return new AttendeeCheckInJob();
           default:
               return null;
       }
   }
}

Implementation of JobCreator

public class AttendeeCheckInJob extends Job {
   ...
   ...
   @NonNull
   @Override
   protected Result onRunJob(Params params) {
       ...
       ...
       Iterable<Attendee> attendees = attendeeRepository.getPendingCheckIns().blockingIterable();
       for (Attendee attendee : attendees) {
           try {
               Attendee toggled = attendeeRepository.toggleAttendeeCheckStatus(attendee).blockingFirst();
               ...
           } catch (Exception exception) {
               ...
               return Result.RESCHEDULE;
           }
       }
       return Result.SUCCESS;
   }

   public static void scheduleJob() {
       new JobRequest.Builder(AttendeeCheckInJob.TAG)
           .setExecutionWindow(1, 5000L)
           .setBackoffCriteria(10000L, JobRequest.BackoffPolicy.EXPONENTIAL)
           .setRequiredNetworkType(JobRequest.NetworkType.CONNECTED)
           .setRequirementsEnforced(true)
           .setPersisted(true)
           .setUpdateCurrent(true)
           .build()
           .schedule();
   }
}

Job class for attendee check in job

To create a Job, these two methods are overridden. onRunJob is where the actual background job is going to run. This is the place where you implement your job logic which should be run in the background. In this method, the attendees with pending sync are fetched from the local database and the network requests are made. On failure, the same job is scheduled again. The process goes on until the job is done. scheduleJob method is where the related setting options are set. This method is used to schedule an incomplete job.

So after this implementation, the workflow described above is changed. Now on attendee is found, it is updated in local database before making any request to the server and the attendee is flagged as pending sync. Accordingly, in the UI single tick is shown for the attendee which is pending for sync with the server. Once the request is made to the server and the response is received, the pending sync flag of the attendee is removed and double tick is shown against the attendee.

Links:
1. Documentation for Android-Job Library by evernote
2. Github Repository of Android-Job Library