Multiple Tickets: Back-end

In my previous post I talked about approach for Multiple Ticket feature’s user-interface [Link]. In this post I’ll discuss about Flask back-end used for saving multiple tickets.

HTML Fields Naming

Since the number of Tickets a user creates is unknown to the server, details of tickets were needed to be sent as an array of values. So the server would accept the list of values and iterate over them. To send data as an array the naming had to include brackets. Below are some input fields used in tickets:

<tr>
    <td>
        <input type="hidden" name="tickets[type]">
        <input type="text" name="tickets[name]" class="form-control" placeholder="Ticket Name" required="required" data-uniqueticket="true">
        <div class="help-block with-errors"></div>
    </td>
    <td>
        <input type="number" min="0" name="tickets[price]" class="form-control"  placeholder="$" value="">
    </td>
    <td>
        <input type="number" min="0" name="tickets[quantity]" class="form-control" placeholder="100" value="{{ quantity }}">
    </td>
    <!-- Other fields -->
</tr>

At the server

When the POST request reaches the server, any of the above fields (say tickets[name]) would be available as a list. The Flask Request object includes a form dictionary that contains all the POST parameters sent with the request. This dictionary is an ImmutableMultiDict object, which has a getlist method to get array of elements.

For instance in our case, we can get tickets[name] using:

@expose('/create', methods=('POST', 'GET'))
def create_view(self):
    if request.method == 'POST':
        ticket_names = request.form.getlist('tickets[name]')

    # other stuff

The ticket_names variable would contain the list of all the Ticket names sent with the request. So for example if the user created three tickets at the client-side, the form would possibly look like:

<form method="post">
  <!-- Ticket One -->
  <input type="text" name="tickets[name]" class="form-control" value="Ticket Name One">
  <!-- Ticket Two -->
  <input type="text" name="tickets[name]" class="form-control" value="Ticket Name Two">
  <!-- Ticket Three -->
  <input type="text" name="tickets[name]" class="form-control" value="Ticket Name Three">

</form>

After a successful POST request to the server, ticket_names should contain ['Ticket Name One', 'Ticket Name Two', 'Ticket Name Three'].

Other fields, like tickets[type], tickets[price], etc. can all be extracted from the Request object.

Checkbox Fields

A problem arose when a checkbox field was needed for every ticket. In my case, a “Hide Ticket” option was needed to let the user decide if he wants the ticket to be shown at the public Events page.

Screenshot from 2016-08-13 12:39:29

The problem with checkboxes is that, for a checkbox of a particular name attribute, if it is not selected, POST parameters of the request made by the client will not contain the checkbox input field parameter. So if I define an input field as a checkbox with the following naming convention, and make a POST request to the server, the server will receive blah[] parameter only if the input element had been checked.

<input type="checkbox" name="blah[]" >

This creates a problem for “Hide ticket” checkboxes. For instance, at the client-side the user creates three tickets with the first and last tickets having their checkboxes selected, the server would get an array of two.

<form>
  <!-- Ticket One -->
  <input type="checkbox" name="tickets[hide]" checked>
  <!-- Ticket Two -->
  <input type="checkbox" name="tickets[hide]">
  <!-- Ticket Three -->
  <input type="checkbox" name="tickets[hide]" checked>

</form>
ticket_hide_opts = request.form.getlist('tickets[hide]')

ticket_hide_opts would be an array of length two. And there is no way to tell what ticket had its “Hide ticket” option checked. So for the hide checkbox field I had to define input elements with unique names to extract them at the server.

There is also a hack to overcome the unchecked-checkbox problem. It is by using a hidden field with the same name as the checkbox. You can read about it here: http://www.alexandrejoseph.com/blog/2015-03-03-flask-unchecked-checkbox-value.html.

Multiple Tickets: User Interface

An Event can have multiple tickets for different purposes. For instance an Arts Exhibition can have multiple Galleries. The Organizer might be interested in assigning a ticket (let’s assume paid) for each Gallery. The user can then buy tickets for the Galleries that he wishes to attend. The feature that Multiple Tickets really provide is exclusiveness. Let’s say Gallery1 has a shorter area (by land) than others. Obviously the Organizer would want fewer people to be present there than other Galleries. To do this, he can create a separate ticket for Gallery1 and specify a shorter sales period. He can also reduce the Maximum number of order that a user can make (max_order). If we would have implemented single ticket per event, this wouldn’t have been possible.

Tickets at Wizard

To handle multiple tickets at the wizard, proper naming of input tags was required. Since the number of tickets that can be created by the user was unknown to the server we had to send ticket field values as lists. Also at the client-side a way was required to let users create multiple tickets.

User Interface

A ticket can be of three types: Free, Paid and Donation. Out of these, only the Paid tickets need a Price. The Tickets holder could be a simple table, with every ticket being a table row. This became more complex afterwards, when more details about the ticket needed to be displayed. A ticket would then be two table rows with one of them (details) hidden.

Ticket holder can be a simple bootstrap table:

<table class="table tickets-table">
    <thead>
        <tr>
            <th>Ticket Name</th>
            <th>Price</th>
            <th>Quantity</th>
            <th>Options</th>
        </tr>
    </thead>
    <tbody>
      <!-- Ticket -->
      <tr>
        <!-- Main info -->
      </tr>
      <tr>
        <!-- More details (initially hidden) -->
      </tr>
      <!-- /Ticket -->
    </tbody>
</table>

To make ticket creation interactive, three buttons were needed to create the above three tickets. The type-name doesn’t not necessarily have to be shown to the user. It could be specified with the Price. For Paid ticket, the Price input element would be a number. For Free and Donation tickets, a Price input element wasn’t required. We could specify an element displaying one of the two types: Free or Donation.

Here’s the holder table with a Free Ticket and a Donation Ticket:

Screenshot from 2016-08-09 16:51:06

Since only the Price field is changing in the three types of tickets, I decided to create a template ticket outside of the form and create a JavaScript function to create one of the tickets by cloning the template.

A Free Ticket with its edit options opened up. You can see other details about the ticket in the second table row.

Screenshot from 2016-08-09 16:52:14

This is a simplified version of the template. I’ve removed common bootstrap elements (grid system) including some other fields.

<div id="ticket-template">
<tr>
    <td>
        <input type="hidden" name="tickets[type]">
        <input type="text" name="tickets[name]" class="form-control" placeholder="Ticket Name" required="required" data-uniqueticket="true">
        <div class="help-block with-errors"></div>
    </td>
    <td>
        <!-- Ticket Price -->
    </td>
    <td>
        <input type="number" min="0" name="tickets[quantity]" class="form-control" placeholder="100" value="{{ quantity }}">
    </td>
    <td>
        <div class="btn-group">
            <a class="btn btn-info edit-ticket-button" data-toggle="tooltip" title="Settings">
                <i class="glyphicon glyphicon-cog"></i>
            </a>
            <a class="btn btn-info remove-ticket-button" data-toggle="tooltip" title="Remove">
                <i class="glyphicon glyphicon-trash"></i>
            </a>
        </div>
    </td>
</tr>
<tr>
    <td colspan="4">
        <div class="row" style="display: none;">
            <!-- Other fields including Description, Sales Start and End time,
              Min and Max Orders, etc.
            -->
        </div>
    </td>
</tr>
</div>

Like I said, the Price element of ticket will make the type obvious for the user, so a type field does not need to be displayed. But the type field is required by the server. You can see it specified as hidden in the template.

The function to create a Ticket according to the type:

I’ve commented snippets to make it easy to understand.

function createTicket(type) {
    /* Clone ticket from template */
    var $tmpl = $("#ticket-template").children().clone();

    var $ticket = $($tmpl[0]).attr("id", "ticket_" + String(ticketsCount));
    var $ticketMore = $($tmpl[1]).attr("id", "ticket-more_" + String(ticketsCount));

    /* Bind datepicker and timepicker to dates and times */
    $ticketMore.find("input.date").datepicker();
    $ticketMore.find("input.time").timepicker({
        'showDuration': true,
        'timeFormat': 'H:i',
        'scrollDefault': 'now'
    });

    /* Bind iCheck to checkboxes */
    $ticketMore.find("input.checkbox.flat").iCheck({
        checkboxClass: 'icheckbox_flat-green',
        radioClass: 'iradio_flat-green'
    });

    /* Bind events to Edit (settings) and Remove buttons */
    var $ticketEdit = $ticket.find(".edit-ticket-button");
    $ticketEdit.tooltip();
    $ticketEdit.on("click", function () {
        $ticketMore.toggle("slow");
        $ticketMore.children().children(".row").slideToggle("slow");
    });

    var $ticketRemove = $ticket.find(".remove-ticket-button");
    $ticketRemove.tooltip();
    $ticketRemove.on("click", function () {
        var confirmRemove = confirm("Are you sure you want to remove the Ticket?");
        if (confirmRemove) {
            $ticket.remove();
            $ticketMore.remove();
        }
    });

    /* Set Ticket Type field */
    $ticket.children("td:nth-child(1)").children().first().val(type);

    /* Set Ticket Price field */
    var html = null;
    if (type === "free") {
        html = '';
    } else if (type === "paid") {
        html = '';
    } else if (type === "donation") {
        html = '';
    }
    $ticket.children("td:nth-child(2)").html(html);

    /* Append ticket to table */
    $ticket.hide();
    $ticketMore.children().children(".row").hide();
    $ticketsTable.append($ticket, $ticketMore);
    $ticket.show("slow");

    ticketsCount += 1;
  }

The flow is simple. Clone the template, bind events to various elements, specify type and price fields and then append to the ticket holder table.

Screenshot from 2016-08-09 17:00:59

We use the Datepicker and Timepicker JavaScript libraries for date and time elements. So fields using these widgets need to have methods called on the elements. Also, we use iCheck for checkboxes and radio buttons. Apart from these, the Edit-Ticket and Remove-Ticket buttons also need event handlers. Edit-Ticket button toggles the Ticket Details segment (second tr of a ticket). Remove-Ticket deletes the ticket. After the Price and Type fields are set, the ticket is appended to the holder table with slow animation.

Flask-SocketIO Notifications

In the previous post I explained about configuring Flask-SocketIO, Nginx and Gunicorn. This post includes integrating Flask-SocketIO library to display notifications to users in real time.

Flask Config

For development we use the default web server that ships with Flask. For this, Flask-SocketIO fallsback to long-polling as its transport mechanism, instead of WebSockets. So to properly test SocketIO I wanted to work directly with Gunicorn (hence the previous post about configuring development environment). Also, not everyone needs to be bothered with the changes required to run it.

class DevelopmentConfig(Config):
    DEVELOPMENT = True
    DEBUG = True

    # If Env Var `INTEGRATE_SOCKETIO` is set to 'true', then integrate SocketIO
    socketio_integration = os.environ.get('INTEGRATE_SOCKETIO')
    if socketio_integration == 'true':
        INTEGRATE_SOCKETIO = True
    else:
        INTEGRATE_SOCKETIO = False

    # Other stuff

SocketIO is integrated (in development env) if the developer has set the INTEGRATE_SOCKETIO environment variable to “true”. In Production, our application runs on Gunicorn, and SocketIO integration must always be there.

Flow

To send message to a particular connection (or a set of connections) Flask-SocketIO provides Rooms. The connections are made to join a room and the message is sent in the room. So to send message to a particular user we need him to join a room, and then send the message in that room. The room name needs to be unique and related to just one user. The User database Ids could be used. I decided to keep user_{id} as the room name for a user with id {id}. This information (room name) would be needed when making the user join a room, so I stored it for every user that logged in.

@expose('/login/', methods=('GET', 'POST'))
    def login_view(self):
        if request.method == 'GET':
            # Render template
        if request.method == 'POST':
            # Take email and password from form and check if 
            # user exists. If he does, log him in.
            login.login_user(user)

            # Store user_id in session for socketio use
            session['user_id'] = login.current_user.id

            # Redirect

After the user logs in, a connection request from the client is sent to the server. With this connection request the connection handler at server makes the user join a room (based on the user_id stored previously).

@socketio.on('connect', namespace='/notifs')
def connect_handler():
    if current_user.is_authenticated():
        user_room = 'user_{}'.format(session['user_id'])
        join_room(user_room)
        emit('response', {'meta': 'WS connected'})

The client side is somewhat similar to this:

<script src="{{ url_for('static', filename='path/to/socket.io-client/socket.io.js') }}"></script>
<script type="text/javascript">
$(document).ready(function() {
    var namespace = '/notifs';

    var socket = io.connect(location.protocol + "//" + location.host + namespace, {reconnection: false});

    socket.on('response', function(msg) {
        console.log(msg.meta);
        // If `msg` is a notification, display it to the user.
    });
});
</script>

Namespaces helps when making multiple connections over the same socket.

So now that the user has joined a room we can send him notifications. The notification data sent to the client should be standard, so the message always has the same format. I defined a get_unread_notifs method for the User class that fetches unread notifications.

class User(db.Model):
    # Other stuff

    def get_unread_notifs(self, reverse=False):
        """Get unread notifications with titles, humanized receiving time
        and Mark-as-read links.
        """
        notifs = []
        unread_notifs = Notification.query.filter_by(user=self, has_read=False)
        for notif in unread_notifs:
            notifs.append({
                'title': notif.title,
                'received_at': humanize.naturaltime(datetime.now() - notif.received_at),
                'mark_read': url_for('profile.mark_notification_as_read', notification_id=notif.id)
            })

        if reverse:
            return list(reversed(notifs))
        else:
            return notifs

This class method is used when a notification is added in the database and has to be pushed into the user SocketIO room.

def create_user_notification(user, action, title, message):
    """
    Create a User Notification
    :param user: User object to send the notification to
    :param action: Action being performed
    :param title: The message title
    :param message: Message
    """
    notification = Notification(user=user,
                                action=action,
                                title=title,
                                message=message,
                                received_at=datetime.now())
    saved = save_to_db(notification, 'User notification saved')

    if saved:
        push_user_notification(user)

def push_user_notification(user):
    """
    Push user notification to user socket connection.
    """
    user_room = 'user_{}'.format(user.id)
    emit('response',
         {'meta': 'New notifications',
          'notif_count': user.get_unread_notif_count(),
          'notifs': user.get_unread_notifs()},
         room=user_room,
         namespace='/notifs')

Permission Decorators

A follow-up to one of my previous posts: Organizer Server Permissions System.

I recently had a requirement to create permission decorators for use in our REST APIs. There had to be separate decorators for Event and Services.

Event Permission Decorators

Understanding Event permissions is simple: Any user can create an event. But access to an event is restricted to users that have Event specific Roles (e.g. Organizer, Co-organizer, etc) for that event. The creator of an event is its Organizer, so he immediately gets access to that event. You can read about these roles in the aforementioned post.

So for Events, create operation does not require any permissions, but read/update/delete operations needed a decorator. This decorator would restrict access to users with event roles.

def can_access(func):
    """Check if User can Read/Update/Delete an Event.
    This is done by checking if the User has a Role in an Event.
    """
    @wraps(func)
    def wrapper(*args, **kwargs):
        user = UserModel.query.get(login.current_user.id)
        event_id = kwargs.get('event_id')
        if not event_id:
            raise ServerError()
        # Check if event exists
        get_object_or_404(EventModel, event_id)
        if user.has_role(event_id):
            return func(*args, **kwargs)
        else:
            raise PermissionDeniedError()
    return wrapper

The has_role(event_id) method of the User class determines if the user has a Role in an event.

# User Model class

    def has_role(self, event_id):
        """Checks if user has any of the Roles at an Event.
        """
        uer = UsersEventsRoles.query.filter_by(user=self, event_id=event_id).first()
        if uer is None:
            return False
        else:
            return True

Reading one particular event (/events/:id [GET]) can be restricted to users, but a GET request to fetch all the events (/events [GET]) should only be available to staff (Admin and Super Admin). So a separate decorator to restrict access to Staff members was needed.

def staff_only(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        user = UserModel.query.get(login.current_user.id)
        if user.is_staff:
            return func(*args, **kwargs)
        else:
            raise PermissionDeniedError()
    return wrapper

Service Permission Decorators

Service Permissions for a user are defined using Event Roles. What Role a user has in an Event determines what Services he has access to in that Event. Access here means permission to Create, Read, Update and Delete services. The User model class has four methods to determine the permissions for a Service in an event.

user.can_create(service, event_id)
user.can_read(service, event_id)
user.can_update(service, event_id)
user.can_delete(service, event_id)

So four decorators were needed to put alongside POST, GET, PUT and DELETE method handlers. I’ve pasted snippet for the can_update decorator. The rest are similar but with their respective permission methods for User class object.

def can_update(DAO):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            user = UserModel.query.get(login.current_user.id)
            event_id = kwargs.get('event_id')
            if not event_id:
                raise ServerError()
            # Check if event exists
            get_object_or_404(EventModel, event_id)
            service_class = DAO.model
            if user.can_update(service_class, event_id):
                return func(*args, **kwargs)
            else:
                raise PermissionDeniedError()
        return wrapper
    return decorator

This decorator is a little different than can_access event decorator in a way that it takes an argument, DAO. DAO is Data Access Object. A DAO includes a database Model and methods to create, read, update and delete object of that model. The db model for a DAO would be the Service class for the object. You can look that the model class is taken from the DAO and used as the service class.

The can_create, can_read and can_delete decorators look exactly the same except they use their (obvious) permission methods on the User class object.

Shell hacks

Custom Shell

Working with database models needs a lot of use of the flask shell. You can access it with:

python manage.py shell

The default shell is quite unintuitive. It doesn’t pretty print the outputs and has no support for auto-completion. I’ve installed IPython that takes care of that. But still working with models means writing a lot of import statements. Plus if there was some change in the code related to the app, then the shell had to restarted again so the changes could be loaded. Meaning writing the import statements again.

We were using Flask-Script and I wanted to run a custom shell that imports all the required modules, models and helper functions, so I don’t have to write them over and over. Some of them were as long as:

from open_event.models.users_events_roles import UsersEventsRoles

So I created a custom shell command with different context that overrides the default shell provided by the Flask-Script Manager. It was pretty easy with Flask-Script. One thing I had to keep in mind is that it needed to be in a different file than manage.py. Since manage.py was committed to source repo, changes to it would be tracked. So I needed a different file that could be excluded from the source repo. I created an smg.py that imported the Manager from the open_event module and overrides the shell command.

from flask_script import Shell

from open_event import manager
from open_event.models.user import User
from open_event.models.event import Event
from open_event.helpers.data import save_to_db, delete_from_db

def _make_context():
    return dict(
        uq=User.query,
        eq=Event.query,
        su=User.query.get(1),
        User=User,
        Event=Event,
        savetodb=save_to_db,
        deletefromdb=delete_from_db
    )


if __name__ == "__main__":
    manager.add_command('shell', Shell(make_context=_make_context))
    manager.run()

Place this smg.py file in the same directory as manage.py, so you can access it with python smg.py shell.

The code is pretty simple to understand. We import the Shell class from flask_script, create its object with our context and then add it to the manager as a command. _make_context contains what I usually like to have in my shell. It must always return a dictionary. The keys of this dictionary would be available as statements inside the shell with their values specified here.

This helps a lot. Most of the time I would be working with the super_admin user, and I would need its User object from time to time. The super_admin user is always going to be the first user (User object with id 1). So instead of from open_event.models.user import User; su = User.query.get(1) I could just use the su variable. Models like User and Event are also readily available (so are their base queries). This is the default context that I always keep, but many times you need more models than the ones specified here. Like when I was working with the permissions system.

from open_event.models.users_events_roles import UsersEventsRoles
from open_event.models.service import Service
from open_event.models.role import Role

def _make_context():
    return dict(
        # usual stuff
        UER=UsersEventsRoles,
        Service=Service,
        Role=Role
    )

You can even write a script that fetches all the database models (instance of sqlalchemy Model class) and then add them to the _make_context dictionary. I like to keep it minimum and differently named, so there are no conflicts of classes when try to auto-complete.

One more thing, you need to exclude the smg.py file so that git doesn’t track it. You can simply add it in the .git/info/exclude file.

I also wrote some useful one-liner bash commands.

Revision History

Every time someone updates the database models, he needs to migrate and provide the migrations file to others by committing it to the source. These files help us upgrade our already defined database. We work with Alembic. Regarding alembic revisions (migration files) you can keep two things in mind. One is the Head, that keeps track of the latest revision, and another is Current, that specifies what revision your tables in the database are based on. If your current revision is not a head, it means your database tables are not up-to-date. You need to upgrade (python manage.py db upgrade). The can be multiple heads in the revision history. python manage.py heads displays all of them. The current revision can be fetched with python manage.py current. I wanted something that automatically checks if current is at a head.

python manage.py db history | grep --color "$(python manage.py db current 2> /dev/null)|$(python manage.py db heads)|$"

You can specify it as an alias in .bashrc. This command displays the revision history with the head and current revision being colored.

Screenshot from 2016-07-11 16:34:18

You can identify the head as it is always appended by “(head)”. The other colored revision would be the current.

You can also reverse the output, by printing bottom to up. This is helpful when the revision history gets large and you have to scroll up to see the head.

alias rev='python manage.py db history | tac | grep --color "$(python manage.py db current 2> /dev/null)|$(python manage.py db heads)|$"'

Screenshot from 2016-07-11 16:42:30.png

Current Revision changer

The current revision is maintained at the database in the one-column alembic_version table. If for some reason the migration file for current revision no longer exists (like when changing to a branch at git that doesn’t have the file), alembic would raise an error. So to change the revision at the database I wrote a bash function.

change_db_current () {
    echo "Current Revision: "
    read current_rev
    echo "New Revision: "
    read new_rev

    echo 'c test \ update alembic_version set version_num='"'"$new_rev"'"' where version_num='"'"$current_rev"'"';' | sudo su - postgres -c psql
}

Enter the current revision hash (displayed in the alembic error) and the revision you need to point the current to, and it will update the table with the new revision. Sick right! It’s pretty stupid actually. It was when I didn’t know how to stamp in alembic. Anyways, it was useful some time ago.

Organizer Server Permissions System

This post discusses about the Organizer Server Permissions System and how it has been implemented using database models.

The Organizer Server Permissions System includes Roles, and Services that these roles can access. Roles can broadly be classified into Event-specific Roles and System-wide Roles.

System-Wide Roles

System-wide roles can be considered a part of the staff maintaining the platform. We define two such roles: Admin and Super Admin. The Super Admin is the highest level user with permissions to access system logs, manage event-specific roles, etc.

System-wide roles are the easiest to implement. Since they’re directly related to a user we can add them as class variables in the User models.

class User(db.Model):
    # other stuff

    is_admin = db.Column(db.Boolean, default=False)
    is_super_admin = db.Column(db.Boolean, default=False)

    @property
    def is_staff(self):
        return self.is_admin or self.is_super_admin

Staff groups Admin and Super Admin roles. So to check if a user is a part of the staff (an admin or a super admin), the is_staff property can directly be used.

user.is_staff

Event-Specific Roles

An Event itself can contain many entities like Tracks, Sponsors, Sessions, etc. Our goal was to define permissions for Roles to access these entities. We grouped these entities and put them under “Services”. Services are nothing but database models associated with an event, that need to have restricted access for the Roles.

We define the following Services for an Event:

  • Track
  • Session
  • Speaker
  • Sponsor
  • Microlocation

Each of these services can either be created, read, updated or deleted. And depending on the Role a user has been assigned for a particular event, he or she can perform such operations.

There are four Event-Specific Roles:

  • Organizer
  • Co-organizer
  • Track Organizer
  • Moderator

As soon as the user creates an event, he is assigned the role of an Organizer, giving him access to all the services for that event. An Organizer can perform any operation on any of the services. A Co-organizer also has access to all the services but can only update them. A Track Organizer can just read and update already created Tracks. The Moderator can only read Tracks.

Although the initial distribution of permissions is kept as above, the Super Admin can (has permissions to) edit them later.

Screenshot from 2016-06-24 02:41:47

To implement permissions for Event specific roles, three new database models were required: Role, Service and Permission.

Role and Service would contain the above mentioned Roles and Services respectively. Permission would contain a Role column, a Service column and four other columns specifying what operation (create/read/update/delete) that Role is allowed to perform on the Service.

The final objective was to define these methods for the User class:

user.can_create(service, event_id)
user.can_read(service, event_id)
user.can_update(service, event_id)
user.can_delete(service, event_id)

Before this we needed a table specifying what Event Roles have been created for an event, and which users have been assigned these roles. The `UsersEventsRoles` model maintained this relationship.

We also needed to check if a user has been assigned a particular role for an event. For this I created a method for each of the roles.

# Event-specific Roles
ORGANIZER = 'organizer'
COORGANIZER = 'coorganizer'
TRACK_ORGANIZER = 'track_organizer'
MODERATOR = 'moderator'

class User(db.Model):
    # other stuff

    def _is_role(self, role_name, event_id):
        role = Role.query.filter_by(name=role_name).first()
        uer = UsersEventsRoles.query.filter_by(user=self,
                                               event_id=event_id,
                                               role=role).first()
        if not uer:
            return False
        else:
            return True

    def is_organizer(self, event_id):
        return self._is_role(ORGANIZER, event_id)

    def is_coorganizer(self, event_id):
        return self._is_role(COORGANIZER, event_id)

    def is_track_organizer(self, event_id):
        return self._is_role(TRACK_ORGANIZER, event_id)

    def is_moderator(self, event_id):
        return self._is_role(MODERATOR, event_id)

Here _is_role helps reduce code redundancy.

Like I said, our final objective was to create methods that determine if a user has permission to perform a particular operation on a service based on the role, I defined the following methods:

    # ...`User` class

    def _has_perm(self, operation, service_class, event_id):
        # Operation names and their corresponding permission in `Permissions`
        operations = {
            'create': 'can_create',
            'read': 'can_read',
            'update': 'can_update',
            'delete': 'can_delete',
        }
        if operation not in operations.keys():
            raise ValueError('No such operation defined')

        try:
            service_name = service_class.get_service_name()
        except AttributeError:
            # If `service_class` does not have `get_service_name()`
            return False

        service = Service.query.filter_by(name=service_name).first()

        uer_querylist = UsersEventsRoles.query.filter_by(user=self,
                                                         event_id=event_id)
        for uer in uer_querylist:
            role = uer.role
            perm = Permission.query.filter_by(role=role,
                                              service=service).first()
            if getattr(perm, operations[operation]):
                return True

        return False

    def can_create(self, service_class, event_id):
        return self._has_perm('create', service_class, event_id)

    def can_read(self, service_class, event_id):
        return self._has_perm('read', service_class, event_id)

    def can_update(self, service_class, event_id):
        return self._has_perm('update', service_class, event_id)

    def can_delete(self, service_class, event_id):
        return self._has_perm('delete', service_class, event_id)

The can_create, can_read, etc. defined in operations are the four columns in the Permission db model. Like _is_role, _has_perm method helps implementing the DRY philosophy.

Errors and Error handlers for REST API

Errors are an essential part of a REST API system. Error instances must follow a particular structure so the client developer can correctly handle them at the client side. We had set a proper error structure at the beginning of creating REST APIs. It’s as follows:

{
    "error": {
        "message": "Error description",
        "code": 400
    }
}

Any error occurring during client server communication would follow the above format. Code is the returned status code and message is a brief description of the error. To raise an error we used an _error_abort() function which was an abstraction over Flask-RESTplus abort(). We defined an error structure inside _error_abort() and passed it to abort().

def _error_abort(code, message):
    error = {
        'code': code,
        'message': message,
    }
    abort(code, error=error)

This method had its limitations. Since the error handlers were not being overidden, only errors raised through _error_abort() had the defined structure. So if an Internal Server error occurred, the returned error response wouldn’t follow the format.

To overcome this, we wrote our own exceptions for errors and created error handlers to handle them. We first made the response structure more detailed, so the client developer can understand what kind of error is being returned.

{
    "error": {
        "code": 400,
        "message": "'name' is a required parameter",
        "status": "INVALID_FIELD",
        "field": "name"
    }
}

The above is an example of a validation error.

The “status” key helps making the error more unique. For example we use 400 status code for both validation errors and invalid service errors (“Service not belonging to said event”). But both have different statuses: “INVALID_FIELD” and “INVALID_SERVICE”.

The “field” key is only useful in the case validation errors, where it names the field that did not pass the validation. For other errors it remains null.

I first documented what kind of errors we would need.

Code Status Field Description
401 NOT_AUTHORIZED null Invalid token or user not authenticated
400 INVALID_FIELD “field_name” Missing or invalid field
400 INVALID_SERVICE null Service ID mentioned in path does not belong to said Event
404 NOT_FOUND null Event or Service not found
403 PERMISSION_DENIED null User role not allowed to perform such action
500 SERVER_ERROR null Internal server error

Next part was creating exception classes for each one these. I created a base error class that extended the python Exception class.

class BaseError(Exception):
    """Base Error Class"""

    def __init__(self, code=400, message='', status='', field=None):
        Exception.__init__(self)
        self.code = code
        self.message = message
        self.status = status
        self.field = field

    def to_dict(self):
        return {'code': self.code,
                'message': self.message,
                'status': self.status,
                'field': self.field, }

The to_dict() method would help when returning the response in error handlers.

I then extended this base class to other error classes. Here are three of them:

class NotFoundError(BaseError):
    def __init__(self, message='Not found'):
        BaseError.__init__(self)
        self.code = 404
        self.message = message
        self.status = 'NOT_FOUND'


class NotAuthorizedError(BaseError):
    def __init__(self, message='Unauthorized'):
        BaseError.__init__(self)
        self.code = 401
        self.message = message
        self.status = 'NOT_AUTHORIZED'


class ValidationError(BaseError):
    def __init__(self, field, message='Invalid field'):
        BaseError.__init__(self)
        self.code = 400
        self.message = message
        self.status = 'INVALID_FIELD'
        self.field = field

I then defined the error handlers for the api:

@api.errorhandler(NotFoundError)
@api.errorhandler(NotAuthorizedError)
@api.errorhandler(ValidationError)
@api.errorhandler(InvalidServiceError)
def handle_error(error):
    return error.to_dict(), getattr(error, 'code')

For overriding the default error handler, Flask-RESTplus let’s you create one with the same decorator, but without passing and argument to it.

@api.errorhandler
def default_error_handler(error):
    """Returns Internal server error"""
    error = ServerError()
    return error.to_dict(), getattr(error, 'code', 500)

I had set the default error to be the internal server error.

class ServerError(BaseError):
    def __init__(self, message='Internal server error'):
        BaseError.__init__(self)
        self.code = 500
        self.message = message
        self.status = 'SERVER_ERROR'

Now raising any of these error classes would activate the error handlers and a proper response would be sent to the client.

Swagger

Swagger is a specification for describing REST APIs. The main aim of Swagger is to provide a REST API definition format that is readable by both machines and humans.

You can think of two entities in a REST API: One the provider of API, and another the client using the API. Swagger essentially covers the gap between them by providing a format that is easy to use by the client and easy for the provider to define.

If not using Swagger, one would most certainly be creating the API first, writing the documentation (human-readable) with it. The client developer would read the documentation and use APIs as required. With Swagger, the specification can be considered as the document itself, helping both the client developer and the provider.

Here’s an example spec I wrote for our GET APIs at Organizer Server: https://gist.github.com/shivamMg/dacada0b45585bcd9cd0fbe4a722eddf

The format, although readable doesn’t really look what a client developer would be asking for.

Remember that the format is machine readable? What does it mean exactly?

Since the Swagger spec is a defined format, the provider can document it and people can write programs that understand the Swagger specification. Swagger itself comes with a set of tools (http://swagger.io/tools/) that use Swagger definitions created by the API provider to create SDKs for the clients to use.

Swagger-UI

One of our most used tools at our server is the Swagger UI (http://swagger.io/swagger-ui/).

It reads an API spec written in Swagger to generate corresponding UI that people can use to explore the APIs. Every API endpoint can have responses and parameters associated with it. For instance our Event endpoint at Server (“/events/:event_id”).

You can see how the UI displays the Model Schema, required parameters and possible response message.

Screenshot from 2016-06-07 19:08:21

Apart from documentation, Swagger UI provides the “Try it out!” tool that lets you make requests to the server for the corresponding API. This feature is incredibly useful for POST requests. No need for long curl commands in the terminal.

Here’s the Swagger config from our demo application: https://open-event.herokuapp.com/api/v2/swagger.json

The Swagger UI for this config can be found at https://open-event.herokuapp.com/api/v2

The UI for the example spec (gist) I linked before can be browsed here.

Swagger-js

https://github.com/swagger-api/swagger-js

Swagger-js is JS library that reads an API spec written in Swagger and provides an interface to the client developer to interact with the API. We will be using Swagger-js for Open Event Webapp.

Here’s an example to show you how it works. Let’s say the following endpoint returns (json) a list of events.

/events

The client developer can create a GET request and render the list with HTML.

$.getJSON("http://example.com/api/v2/events", function(data) {

  var events = "";
  $.each(data, function(i, event) {
    events += "<li id='event_" + i + "'>" + event.name + "</li>";
  });

  $("ul#events").html(events);
});

What if the provider moves the endpoint to “/event/all”? The client developer would need to change every instance of the URL to http://example.com/event/all.

Let’s now take the case of Swagger-js. With Swagger-js the client developer would essentially be writing programs to interact with the API Swagger spec defined by the provider, instead of directly consuming the API.

window.client = new SwaggerClient({
    url: "http://example.com/api/v2/swagger.json",
    success: function() {
      client.event.getEvents({
        responseContentType: 'application/json'
      }, function(data) {

        /* Create `events` string same as before */

        $("ul#events").html(events);
      });
    }
  });

 

The APIs at the Organizer server are getting more complex. Swagger helps in keeping the spec well defined.

That’s all for now. Hope you enjoyed the read.

Organizer Server and REST APIs

The Open Event Organizer Server is a server application written in Flask. It provides an admin interface for the organizers of events to manage events and related services like Sessions, Tracks, etc. Additionally it provides GET APIs for developers to read data from the server. These APIs are consumed by the Open Event Webapp and Android Client to display details to the users. The existing APIs could only fetch data. My is to create REST APIs that write data to the server.

Developers are divided into groups with every group handling one aspect of the project. Avi Aryan and I would be working on REST APIs. Currently I’m working on porting the existing GET APIs to newer spec. Justin decided on our stack and technologies we would be working with (link). We are using Flask-Restplus extension for building APIs.

The write API is not going to be the only big change to the server. The new authorization system is also going to change a lot. It would include more user roles, with each of them having different set of permissions. Apart from the Administrator and Organizer, a user can be:

  • Attendee
  • Moderator
  • Speaker
  • Track Organizer

Other changes to User Management are mentioned in the docs.

Besides these, adding support for OAuth 2.0 is also on the list. This would let users sign up through Social Media platforms.

I had thought of a possible work flow for my group.

  1. Port existing GET APIs with newer spec on Flask-Restplus.
  2. Add the user roles mentioned above to the authorization system. A user role defines what type of services a user has access to. For example, an Attendee has write access to feedback and rating system for Sessions but he does not have permissions to create or modify Tracks. This needs to be done before creating write (POST/PUT/DELETE) APIs, so that access to services can be defined according to the user roles. The GET APIs are public and do not require any such permissions.
  3. Create write APIs.
  4. Set up user authentication system to register and sign in users.
  5. Add support for OAuth 2.0.

Many of these changes require other changes to the server. Like the feedback and rating system for Attendees has not been implemented yet and would first require definitions for its database models. Existing models would also require changes. A lot of work to do.

I have previously worked with Python and Django. This project brings a lot of new stuff for me. Python2, Flask, Swagger and RESTful APIs. DukeJustin and Mario are going to be mentors for the Organizer Server. All in all, this summer is going to be an exciting one 😀.

About Me:

I’m Shivam, a 3rd year Computer Engineering student from College of Technology, Pantnagar. This is my first attempt at GSOC and I’m grateful to FOSSASIA for accepting my proposal. I started out with FOSSASIA by contributing to their Open Source projects.