The Open Event Ecosystem

This post contains data collected and compiled from various sources that include (but not limited to) project readme files, my presentation from FOSSASIA 17, Wikipedia, my head.

This aims to be a place for any new contributor to know the basics about the entire Open Event Project.

This could also help newcomers choose an Open Event sub-project of their liking and start contributing .

The Open Event Project offers event managers a platform to organise all kinds of events including concerts, conferences, summits and regular meet-ups. The components support organisers in all stages from event planning to publishing, marketing and ticket sales. Automated web and mobile apps help attendees to get information easily.

There are seven components of the project:

  • Open Event Universal Format – A standard JSON Schema specification for data sharing/transfer/import/export between various components of Open Event.
  • Open Event API Server – The core of the project. The API Server and database.
  • Open Event Frontend – The primary frontend for the Open Event API Server where organisers, speakers and attendees can sign-up and perform various functions.
  • Open Event Organiser App – A cross-platform mobile application for Organisers with ticketing, checking and quick management capabilities.
  • Open Event Android application generator – An android application generator that allows event organisers to generate customised android applications for their events.
  • Open Event Web application generator – A web application generator that allows event organisers to generate customised static easily-hostable web pages for their events.
  • Open Event Data scrappers – A scrapper that allows users to scrape events from popular websites such as eventbrite into the Open Event Universal Format for it to be imported into Open Event.

Open Event Universal Format

A standard JSON Schema specification for data sharing/transfer/import/export between various components of Open Event.

Repository: fossasia/open-event.

Open Event API Server

The core of the project. The API Server and database.

Repository: fossasia/open-event-orga-server.

The system exposes a RESTful API to fetch and modify data. It also provides endpoints that allow organisers to import and export event data in a standard compressed file format that includes the event data in JSON (Open Event Universal Format) and binary media files like images and audio.

The Open Event API Server comprises of:

  • Flask web framework – Flask is a microframework for python to create web applications. Flask also provided us with a Jinja2 templating engine.
  • PostgreSQL – Our database. PostgreSQL is an open-sourced Object-relational database management system (ORDBMS). We use SQLAlchemy ORM to communicate with the database and perform database operations.
  • Celery – Celery is a Distributed Task Queue. It allows us to run time consuming and/or resource intensive tasks in the background (or) on a separate worker server. We use celery to process event import/export tasks to process email send requests.
  • Redis – Redis is an in-memory data structure store. It’s generally used for caching and for message brokering b/w different services due it’s insanely fast read-write speeds (since it’s an in-memory data store). We use it for caching results of time-consuming less volatile database calls and also for brokering jobs and their statuses b/w the web server and Celery task queue.

In the near future, we plan to implement more additional components too.

  • Elasticsearch – a distributed, RESTful search and analytics engine. To start with, we’ll be using it to index our events and provide much fast search results to the user.
  • Kibana – data visualization and analytics plugin for elasticsearch. It will allow us to better understand search trends, user demographics and help us provide a better user experience.

We’re now in the process of slowly phasing out the Open Event Server’s existing UI and keep it only as an API Server since we’re moving towards an API-Centric approach with a Fresh new Open Event Frontend.

The Open Event server’s repository contains multiple branches each serving a specific purpose.

  • development – All development goes on in this branch. If you’re making a contribution, please make a pull request to development. PRs to must pass a build check and a unit-test check on Travis (https://open-event-dev.herokuapp.com – Is running off the development branch. It is hosted on Heroku.). This is probably the most unstable branch of all.
  • master – This contains shipped code. After significant features/bug-fixes are accumulated on development, we make a version update, and make a release. (https://eventyay.com – Is running off the master branch. (whichever is the latest release.) Hosted on Google Cloud Platform (Google Container Engine + Kubernetes).
  • staging – This branch is mainly for testing eventyay server configurations on a staging server without affecting the production environment. Push access is restricted to the devops team.
  • gh-pages – This contains the documentation website on http://dev.eventyay.com. The site is build automatically on each commit in the development branch through a script and using travis. It includes the md files of the Readme and /docs folder.

The Open Event Server has Unit tests to ensure each part of the server works as expected. This is also part of our workflow. So, every PR made to the repository, is tested by running the Unit tests and the code coverage is reported on the PR itself. (We use travis and codecov for continuous integration and code coverage reporting respectively).

Open Event Frontend

The primary frontend for the Open Event API Server where organisers, speakers and attendees can sign-up and perform various functions.

Repository: fossasia/open-event-frontend.

Open Event frontend is built on Ember.js – A javascript web application framework. Ember.js uses Ember data – its data persistence module to communicate with the Open Event API Server via the exposed RESTful endpoints.

The frontend is built upon components that are re-usable across various parts of the application. This reduces code duplication and increases maintainability.

The frontend also has various services to share data and functionality b/w components/routes/controllers.

Each merge into the Open Event frontend repository triggers two deployments:

Currently, both the staging and development use the development branch since the frontend is still under active development. Once we reach the release stage, staging & production deployments will start using the master branch just like the Open Event API Server.

As a part of the development workflow, to ensure proper code-styles throughout the project, and to prevent regressions, the project has Acceptance tests, Integration tests, Unit tests and lint checks on both javascript (*.js) and handlebar files (*.hbs). These tests are run for every PR made to the repository. (We use travis and codecov for continuous integration and code coverage reporting respectively).

Open Event Organizer App

A cross-platform mobile application for Organiser with ticketing, checking and quick management capabilities.

Repository: fossasia/open-event-orga-app.

The organiser application is a mobile application that can run on Android, iOS and Windows Phone. It is built using Ionic Framework – cross-platform app development framework. The app uses the RESTful APIs exposed by the Open Event API Server to get data and perform operations.

The core features of this application are

  • Scan a QR code from an attendee’s ticket to view information about the attendee and to check him/her in.
  • Check-in attendees (Attendees can be searched for using name and/or email)
  • Continuous data sync with the Open Event Organiser Server

Other planned feature include:

  • Overview of sales – The organisers can see how their event is performing with just a few taps
  • Overview of tracks and sessions – They can see what sessions are next and can use that information to (for example) get everything ready for that session.
  • Quick session re-scheduling – They can re-schedule sessions if required. This should also trigger notification to participant that have registered for that event.
  • Push notifications for certain triggers – (for example) Organisers can get notifications if any session proposal is received.

This project has only one development branch (master). Each commit to that branch triggers an apk deployment to the apk branch via Travis.

Open Event Android Application Generator

An android application generator that allows event organisers to generate customised android applications for their events.

Repository: fossasia/open-event-android.

This project consists of two components.

  • The App Generator – A web application that is hosted on a server and generates an event Android app from a zip with JSON and binary files (examples here) or through an API.
  • A generic Android app – the output of the app generator. The mobile app can be installed on any Android device for browsing information about the event. Updates can be made automatically through API endpoint connections from an online source (e.g. server), which needs to defined in the provided event zip with the JSON files. The Android app has a standard configuration file, that sets the details of the app (e.g. color scheme, logo of event, link to JSON app data).

  • Flask web framework – Flask is a microframework for python to create web applications. Flask also provided us with a Jinja2 templating engine.
  • Celery – Celery is a Distributed Task Queue. It allows us to run time consuming and/or resource intensive tasks in the background (or) on a separate worker server. The android application compile and build process is run as a celery job. This also allows multiple users to simultaneously use the generator.
  • Redis – Redis is an in-memory data structure store. It’s generally used for caching and for message brokering b/w different services due it’s insanely fast read-write speeds (since it’s an in-memory data store). We use it for brokering jobs and their statuses b/w the web server and Celery task queue.
  • Java Development Kit (JDK) – Java Development Kit provides us with a set of tools consisting of (but not limited to) a compiler, runtime environment, loader which enables us to compiler, build and run java based applications.
  • Android SDK Tools – The Android SDK Toolset provides us with Android API libraries, debugging tools, emulation capabilities and other tools that are needed to develop, build and run java based android applications.
  • Gradle Build Tool – Gradle is an open source build automation system. It allows developers to define a build process as a set of tasks that can be easily executed on any machine with predictable outputs as long as the gradle files are available.

As with other projects, this also has multiple branches each serving a different purpose

  • development – All development goes on in this branch. If you’re making a contribution, you are supposed to make a pull request to development. PRs to master must pass a build check and a unit-test (app/src/test) check on Travis.
  • master – This contains shipped code. After significant features/bugfixes are accumulated on development, we make a version update, and make a release. All tagged commits on master branch will automatically generate a release on Github with a copy of fDroid-debug and GooglePlay-debug apks.
  • apk – This branch contains two apk’s, that are automatically generated on merged pull request a) from the dev branch and b) from the master branch using the Open Event sample of the FOSSASIA Summit. This branch also assists in deploying the generator to http://droidgen.eventyay.com by triggering a travis build every time an apk is pushed to this branch. The reason this type of a round-about way was implemented is that, travis doesn’t allow android and docker builds on the same branch. So, we’re forced to use the apk branch to do the docker build for us.

Open Event Web application generator

A web application generator that allows event organisers to generate customised static easily-hostable web pages for their events.

Repository: fossasia/open-event-webapp.

The Open Event Web App project has two components :

  • An event website generator – A web application that generates the event website based on event data provided by the user either in the form of an archive as per the Open Event Universal Format or an API Endpoint to an Open Event API Server instance
  • A generic web application – This will be customised and additional pages will be generated from a set of templates by the generator based on the provided event data. The generated website bundle can be hosted on any static hosting provider (Github pages, any simple web hosting service, or any web server that can serve HTML files etc).

The generator also gives you the ability to directly upload the generated files to any server (via FTP) or to the gh-pages branch of any repository on GitHub.

  • Express.js – A web application framework for Node.js. The web application generator’s user-facing frontend runs on Express.js framework.
  • Socket.IO – A javascript library for real-time web applications. It allows a client and a server to communicate with each other simultaneously in a real-time manner. (Confusing ? If you wanna build something like a chat-room … you’d need something like Socket.IO). The web application generator uses Socket.IO to handle multiple clients and also to show the correct progress of the website generation to each of those clients in a real-time manner.
  • The web generator also uses SASS – which adds awesome features (Superpowers as their site says) to good-old CS, and Handlebars – which is a templating engine that let’s you easily generate HTML output based on a templates from a given javascript object.

The branches are similar to other projects under Open Event.

  • development – All development goes on in this branch. If you’re making a contribution, you are supposed to make a pull request to development. PRs to master must pass a build check and a unit-test (test/serverTest.js) check on Travis. Gets deployed automatically to https://opev-webgen-dev.herokuapp.com .
  • master – This contains shipped code. After significant features/bugfixes are accumulated on development, we make a version update, and make a release.

Open Event Data Scrappers

A scrapper that allows users to scrape events from popular websites such as eventbrite into the Open Event Universal Format for it to be imported into Open Event.

Repository: fossasia/event-collect.

As of now, only eventbrite is supported for data scrapping. This allows new users of Open Event to import their events from other ticketing/event management websites very easily. The scrapper accepts a query from the user, and gets the event page for that query on eventbrite, parses the HTML DOM to get the required data and compiles everything into the Open Event Universal Format so that it could be imported into Open Event.

The scrapper is written in Python and uses Beautiful Soup to parse the DOM.

Future plans include combining fossasia/query-server, Open Event server and event collect to enable automatic import of events from other websites based on user searches on eventyay.


The Open Event Ecosystem is vast and has plenty of contribution opportunities to both beginners and experts alike. Starting with contributions is as easy as picking a project, choosing an issue and submitting a PR. No strings attached. (Oh wait. there is just one string attached. Ensure you read & follow the “Best Practices at FOSSASIA Guide”)

{ Repost from my personal blog @ https://blog.codezero.xyz/open-event-ecosystem }

Generating xCal calendar in python

{ Repost from my personal blog @ https://blog.codezero.xyz/generate-xcal-calendar-in-python }

“xCal”, is an XML format for iCalendar data.

The iCalendar data format (RFC5545) is a widely deployed interchange format for calendaring and scheduling data.

A Sample xCal document

<?xml version="1.0" encoding="utf-8"?>  
<iCalendar xmlns:xCal="urn:ietf:params:xml:ns:xcal">  
    <vcalendar>
        <version>2.0</version>
        <prodid>-//Pentabarf//Schedule 1.0//EN</prodid>
        <x-wr-caldesc>FOSDEM 2016</x-wr-caldesc>
        <x-wr-calname>Schedule for events at FOSDEM 2016</x-wr-calname>
        <vevent>
            <method>PUBLISH</method>
            <uid>123e4567-e89b-12d3-a456-426655440000</uid>
            <dtstart>20160131T090000</dtstart>
            <dtend>20160131T091000</dtend>
            <duration>00:10:00:00</duration>
            <summary>Introduction to the SDR Track- Speakers, Topics, Algorithm</summary>
            <description>&lt;p&gt;The opening talk for the SDR devroom at FOSDEM 2016.&lt;/p&gt;</description>
            <class>PUBLIC</class>
            <status>CONFIRMED</status>
            <categories>Software Defined Radio</categories>
            <url>https:/fosdem.org/2016/schedule/event/sdrintro/</url>
            <location>AW1.125</location>
            <attendee>Martin Braun</attendee>
        </vevent>
    </vcalendar>
</iCalendar>

Each event/session will be in a seperate vevent block. Each speaker/attendee of an event/session will be in an attendee block inside a vevent block.

Some important elements are:

  1. version – Has the version of the iCalendar data
  2. prodid – Contains the name of the application/generator that generated this document
  3. x-wr-caldesc – A descriptive name for this calendar
  4. x-wr-calname – A description of the calendar

The structure and keywords used in xCal are the same as those used in the iCal format. To generate the XML document, we’ll be using python’s ElementTreeXML API that is part of the Python standard library.

We’ll be using two main classes of the ElementTree API:

  1. Element – used to create a standard node. (Used for the root node)
  2. SubElement – used to create a sub element and attache the new node to a parent

Let’s start with the root iCalendar node and set the required attributes.

from xml.etree.ElementTree import Element, SubElement, tostring

i_calendar_node = Element('iCalendar')  
i_calendar_node.set('xmlns:xCal', 'urn:ietf:params:xml:ns:xcal')

Now, to add the vcalendar node to the iCalendar node.

v_calendar_node = SubElement(i_calendar_node, 'vcalendar')

Let’s add the other aspects of the calendar to the vcalendar node as separate sub nodes.

version_node = SubElement(v_calendar_node, 'version')  
version_node.text = '2.0'

prod_id_node = SubElement(v_calendar_node, 'prodid')  
prod_id_node.text = '-//fossasia//open-event//EN'

cal_desc_node = SubElement(v_calendar_node, 'x-wr-caldesc')  
cal_desc_node.text = "Calendar"

cal_name_node = SubElement(v_calendar_node, 'x-wr-calname')  
cal_name_node.text = "Schedule for sessions"

Now, we have added information about our calendar. Now to add the actual events to the calendar. Each event would be a vevent node, a child of vcalendar node. We can loop through all our available event/sessions and add them to the calendar.

for session in sessions:  
    v_event_node = SubElement(v_calendar_node, 'vevent')

    uid_node = SubElement(v_event_node, 'uid')
    uid_node.text = str(session.id)

    dtstart_node = SubElement(v_event_node, 'dtstart')
    dtstart_node.text = session.start_time.isoformat()

    dtend_node = SubElement(v_event_node, 'dtend')
    dtend_node.text = tz.localize(session.end_time).isoformat()

    duration_node = SubElement(v_event_node, 'duration')
    duration_node.text =  "00:30"

    summary_node = SubElement(v_event_node, 'summary')
    summary_node.text = session.title

    description_node = SubElement(v_event_node, 'description')
    description_node.text = session.short_abstract

    class_node = SubElement(v_event_node, 'class')
    class_node.text = 'PUBLIC'

    status_node = SubElement(v_event_node, 'status')
    status_node.text = 'CONFIRMED'

    categories_node = SubElement(v_event_node, 'categories')
    categories_node.text = session.session_type.name

    url_node = SubElement(v_event_node, 'url')
    url_node.text = "https://some.conf/event/" + str(session.id)

    location_node = SubElement(v_event_node, 'location')
    location_node.text = session.microlocation.name

    for speaker in session.speakers:
        attendee_node = SubElement(v_event_node, 'attendee')
        attendee_node.text = speaker.name

Please note that all the timings in the XML Document must comply with ISO 8601 and must have the date+time+timezone. Example: 2007-04-05T12:30-02:00.

We’re still not done yet. We now have the XML document as an Element object. But we’ll be needing it as a string to either store it somewhere or display it.

The document can be converted to a string by using the ElementTree API’s tostring helper method and passing the root node.

xml_as_string = tostring(i_calendar_node)

And that’s it. You now have a proper XML document representing your events.

Building the Scheduler UI

{ Repost from my personal blog @ https://blog.codezero.xyz/building-the-scheduler-ui }

If you hadn’t already noticed, Open Event has got a shiny new feature. A graphical and an Interactive scheduler to organize sessions into their respective rooms and timings.

As you can see in the above screenshot, we have a timeline on the left. And a lot of session boxes to it’s right. All the boxes are re-sizable and drag-drop-able. The columns represent the different rooms (a.k.a micro-locations). The sessions can be dropped into their respective rooms. Above the timeline, is a toolbar that controls the date. The timeline can be changed for each date by clicking on the respective date button.

The Clear overlaps button would automatically check the timeline and remove any sessions that are overlapping each other. The Removed sessions will be moved to the unscheduled sessions pane at the left.

The Add new micro-location button can be used to instantly add a new room. A modal dialog would open and the micro-location will be instantly added to the timeline once saved.

The Export as iCal allows the organizer to export all the sessions of that event in the popular iCalendar format which can then be imported into various calendar applications.

The Export as PNG saves the entire timeline as a PNG image file. Which can then be printed by the organizers or circulated via other means if necessary.

Core Dependencies

The scheduler makes use of some javascript libraries for the implementation of most of the core functionality

  • Interact.js – For drag-and-drop and resizing
  • Lodash – For array/object manipulations and object cloning
  • jQuery – For DOM Manipulation
  • Moment.js – For date time parsing and calculation
  • Swagger JS – For communicating with our API that is documented according to the swagger specs.
Retrieving data via the API

The swagger js client is used to obtain the sessions data using the API. The client is asynchronously initialized on page load. The client can be accessed from anywhere using the javascript function initializeSwaggerClient.

The swagger initialization function accepts a callback which is called if the client is initialized. If the client is not initialized, the callback is called after that.

var swaggerConfigUrl = window.location.protocol + "//" + window.location.host + "/api/v2/swagger.json";  
window.swagger_loaded = false;  
function initializeSwaggerClient(callback) {  
    if (!window.swagger_loaded) {
        window.api = new SwaggerClient({
            url: swaggerConfigUrl,
            success: function () {
                window.swagger_loaded = true;
                if (callback) {
                    callback();
                }

            }
        });
    } else {
        if (callback) {
            callback();
        }
    }
}

For getting all the sessions of an event, we can do,

initializeSwaggerClient(function () {  
    api.sessions.get_session_list({event_id: eventId}, function (sessionData) {
        var sessions = sessionData.obj;  // Here we have an array of session objects
    });
});

In a similar fashion, all the micro-locations of an event can also be loaded.

Processing the sessions and micro-locations

Each session object is looped through, it’s start time and end time are parsed into moment objects, duration is calculated, and it’s distance from the top in the timeline is calculated in pixels. The new object with additional information, is stored in an in-memory data store, with the date of the event as key, for use in the timeline.

The time configuration is specified in a separate time object.

var time = {  
    start: {
        hours: 0,
        minutes: 0
    },
    end: {
        hours: 23,
        minutes: 59
    },
    unit: {
        minutes: 15,
        pixels: 48,
        count: 0
    },
    format: "YYYY-MM-DD HH:mm:ss"
};

The smallest unit of measurement is 15 minutes and 48px === 15 minutes in the timeline.

Each day of the event is stored in a separate array in the form of Do MMMM YYYY(eg. 2nd May 2013).

The array of micro-location objects is sorted alphabetically by the room name.

Displaying sessions and micro-locations on the timeline

According to the day selected, the sessions for that day are displayed on the timeline. Based on their time, the distance of the session div from the top of the timeline is calculated in pixels and the session box is positioned absolutely. The height of the session in pixels is calculated from it’s duration and set.

For pixels-minutes conversion, the following are used.

/**
 * Convert minutes to pixels based on the time unit configuration
 * @param {number} minutes The minutes that need to be converted to pixels
 * @returns {number} The pixels
 */
function minutesToPixels(minutes) {  
    minutes = Math.abs(minutes);
    return (minutes / time.unit.minutes) * time.unit.pixels;
}

/**
 * Convert pixels to minutes based on the time unit configuration
 * @param {number} pixels The pixels that need to be converted to minutes
 * @returns {number} The minutes
 */
function pixelsToMinutes(pixels) {  
    pixels = Math.abs(pixels);
    return (pixels / time.unit.pixels) * time.unit.minutes;
}
Adding interactivity to the session elements

Interact.js is used to provide interactive capabilities such as drag-drop and resizing.

To know how to use Interact.js, you can checkout some previous blog posts on the same, Interact.js + drag-drop and Interact.js + resizing.

Updating the session information in database on every change

We have to update the session information in database whenever it is moved or resized. Every time a session is moved or resized, a jQuery event is triggered on $(document) along with the session object as the payload.

We listen to this event, and make an API request with the new session object to update the session information in the database.


The scheduler UI is more complex than said in this blog post. To know more about it, you can checkout the scheduler’s javascript code atapp/static/js/admin/event/scheduler.js.

Accepting Stripe payments on behalf of a third-party

{ Repost from my personal blog @ https://blog.codezero.xyz/accepting-stripe-payments-on-behalf-of-a-third-party }

In Open Event, we allow the organizer of each event to link their Stripe account, so that all ticket payments go directly into their account. To make it simpler for the organizer to setup the link, we have a Connect with stripe button on the event creation form.

Clicking on the button, the organizer is greeted with a signup flow similar to Login with Facebook or any other social login. Through this process, we’re able to securely and easily obtain the credentials required to accept payments on behalf of the organizer.

For this very purpose, stripe provides us with an OAuth interface called as Stripe Connect. Stripe Connect allows us to connect and interact with other stripe accounts through an API.

We’ll be using Python’s requests library for making all the HTTP Requests to the API.
You will be needing a stripe account for this.

Registering your platform
The OAuth Flow

The OAuth flow is similar to most platforms.

  • The user is redirected to an authorization page where they login to their stripe account and authorize your app to access their account
  • The user is then redirected back to a callback URL with an Authorization code
  • The server makes a request to the Token API with the Authorization code to retrieve the access_token, refresh_token and other credentials.

Implementing the flow

Redirect the user to the Authorization URL.
https://connect.stripe.com/oauth/authorize?response_type=code&client_id=ca_8x1ebxrl8eOwOSqRTVLUJkWtcfP92YJE&scope=read_write&redirect_uri=http://localhost/stripe/callback  

The authorization url accepts the following parameters.

  1. client_id – The client ID acquired when registering your platform.required.
  2. response_type – Response type. The value is always code. required.
  3. redirect_uri – The URL to redirect the customer to after authorization.
  4. scope – Can be read_write or read_only. The default is read_only. For analytics purposes, read_only is appropriate; To perform charges on behalf of the connected user, We will need to request read_write scope instead.

The user will be taken to stripe authorization page, where the user can login to an existing account or create a new account without breaking the flow. Once the user has authorized the application, he/she is taken back to the Callback URL with the result.

Requesting the access token with the authorization code

The user is redirected back to the callback URL.

If the authorization failed, the callback URL has a query string parameter error with the error name and a parameter error_description with the description of the error.

If the authorization was a success, the callback URL has the authorization code in the code query string parameter.

import requests

data = {  
    'client_secret': 'CLIENT_SECRET',
    'code': 'AUTHORIZATION_CODE',
    'grant_type': 'authorization_code'
}

response = requests.post('https://connect.stripe.com/oauth/token', data=data)

The client_secret is also obtained when registering your platform. The codeparameter is the authorization code.

On making this request, a json response will be returned.

If the request was a success, the following response will be obtained.

{
  "token_type": "bearer",
  "stripe_publishable_key": PUBLISHABLE_KEY,
  "scope": "read_write",
  "livemode": false,
  "stripe_user_id": USER_ID,
  "refresh_token": REFRESH_TOKEN,
  "access_token": ACCESS_TOKEN
}

If the request failed for some reason, an error will be returned.

{
  "error": "invalid_grant",
  "error_description": "Authorization code does not exist: AUTHORIZATION_CODE"
}

The access_token token obtained can be used as the secret key to accept payments like discussed in Integrating Stripe in the Flask web framework.

PayPal Express Checkout in Python

As per the PayPal documentation …

Express Checkout is a fast, easy way for buyers to pay with PayPal. Express Checkout eliminates one of the major causes of checkout abandonment by giving buyers all the transaction details at once, including order details, shipping options, insurance choices, and tax totals.

The basic steps for using express checkout to receive one-time payments are:

  1. Getting the PayPal API credentials.
  2. Making a request to the API with the transaction details to get a token
  3. Using the token to send the users to the PayPal payment page
  4. Capturing the payment and charging the user after the user completes the payment at PayPal.

We will be using PayPal’s Classic NVP (Name-value pair) API for implementing this.

Getting PayPal API Credentials

To begin with, we’ll need API Credentials.
We’ll be using the Signature API credentials which consists of

  • API Username
  • API Password
  • Signature

To obtain these, you can follow the steps at Creating and managing NVP/SOAP API credentials – PayPal Developer.

You’ll be getting two sets of credentials. Sandbox and Live. We’ll just stick to the Sandbox for now.

Now, we need sandbox test accounts for making and receiving payments. Head over to Creating Sandbox Test Accounts – PayPal Developer and create two sandbox test accounts. One would be the facilitator and one would be the buyer.

PayPal NVP Servers

All the API actions will take place by making a request to the PayPal server. PayPal has 4 different NVP servers for 4 different purposes.

  1. https://api-3t.sandbox.paypal.com/nvp – Sandbox “testing” server for use with API signature credentials.
  2. https://api-3t.paypal.com/nvp– PayPal “live” production server for use with API signature credentials.
  3. https://api.sandbox.paypal.com/nvp – Sandbox “testing” server for use with API certificate credentials.
  4. https://api.paypal.com/nvp – PayPal “live” production server for use with API certificate credentials.

We’ll be using the Sandbox “testing” server for use with API signature credentials.

Creating a transaction and obtaining the token

To create a transaction, we’ll need to make a request with all the transaction details. We can use Python requests library to easily make the requests. All requests are POST.

We’ll be calling the SetExpressCheckout method of the NVP API to obtain the token.

import requests  
import urlparse

data = {  
    'USER': credentials['USER'],
    'PWD': credentials['PWD'],
    'SIGNATURE': credentials['SIGNATURE'],
    'SUBJECT': credentials['FACILITATOR_EMAIL'],
    'METHOD': 'SetExpressCheckout',
    'VERSION': 93,
    'PAYMENTREQUEST_0_PAYMENTACTION': 'SALE',
    'PAYMENTREQUEST_0_AMT': 100,
    'PAYMENTREQUEST_0_CURRENCYCODE': 'USD',
    'RETURNURL': 'http://localhost:5000/paypal/return/',
    'CANCELURL': 'http://localhost:5000/paypal/cancel/'
}
response = requests.post('https://api-3t.sandbox.paypal.com/nvp', data=data)  
token = dict(urlparse.parse_qsl(response.text))['TOKEN']

Here,

  • USER represents your Sandbox API Username.
  • PWD represents your Sanbox API Password.
  • SIGNATURE represents your Sandbox Signature.
  • SUBJECT represents the facilitator’s email ID.
  • PAYMENTREQUEST_0_AMT is the total transaction amount.
  • PAYMENTREQUEST_0_CURRENCYCODE is the 3 digit ISO 4217 Currency code.
  • RETURNURL is where the user will be sent to after the transaction
  • CANCELURL is where the user will be sent to if he/she cancels the transaction.

A URL-Encoded, Name-value pair response would be obtained. We can decode that into a dict by using Python’s urlparse modules.

From the response, we’re extracting the TOKEN which we will use to generate the payment URL for the user.

This token has to be retained since we’ll be using it in further steps of the process.

Redirecting the user to PayPal for Approval

With the token we obtained, we can form the payment URL.

https://www.sandbox.paypal.com/cgi-bin/webscr?cmd=_express-checkout&token=<TOKEN>

We’ll have to send the user to that URL. Once the user completes the transaction at PayPal, he/she will be returned to the RETURNURL where we’ll further process the transaction.

Obtaining approved payment details and capturing the payment

Once the user completes the transaction and gets redirected back to RETURNURL, we’ll have to obtain the confirmed payment details from PayPal. For that we can again use the token ID that we obtained before.

We’ll now be making a request to the GetExpressCheckoutDetails method of the API.

import requests  
import urlparse

data = {  
    'USER': credentials['USER'],
    'PWD': credentials['PWD'],
    'SIGNATURE': credentials['SIGNATURE'],
    'SUBJECT': credentials['FACILITATOR_EMAIL'],
    'METHOD': 'GetExpressCheckoutDetails',
    'VERSION': 93,
    'TOKEN': TOKEN
}

response = requests.post('https://api-3t.sandbox.paypal.com/nvp', data=data)  
result = dict(urlparse.parse_qsl(response.text))  
payerID = result['PAYERID']

A URL-Encoded, Name-value pair response would be obtained. We can decode that into a dict by using Python’s urlparse modules.

This will provide us with information about the transaction such as transaction time, transaction amount, charges, transaction mode, etc.

But, we’re more interested in the PAYERID which we’ll need to capture/collect the payment. The money is not transferred to the facilitators account until it is captured/collected. So, be sure to collect it.

To collect it, we’ll be making another request to the DoExpressCheckoutPaymentmethod of the API using the token and the PAYERID.

import requests  
import urlparse

data = {  
    'USER': credentials['USER'],
    'PWD': credentials['PWD'],
    'SIGNATURE': credentials['SIGNATURE'],
    'SUBJECT': credentials['FACILITATOR_EMAIL'],
    'METHOD': 'DoExpressCheckoutPayment',
    'VERSION': 93,
    'TOKEN': TOKEN,
    'PAYERID': payerID,
    'PAYMENTREQUEST_0_PAYMENTACTION': 'SALE',
    'PAYMENTREQUEST_0_AMT': 100,
    'PAYMENTREQUEST_0_CURRENCYCODE': 'USD',
}

response = requests.post('https://api-3t.sandbox.paypal.com/nvp', data=data)  
result = dict(urlparse.parse_qsl(response.text))  
status = result['ACK']

All the details have to be the same as the ones provided while obtaining the token. Once we make the request, we’ll again get a URL-Encoded, Name-value pair response. We can decode that into a dict by using Python’s urlparsemodules.

From the response, ACK (Acknowledgement status) will provide us with the status of the payment.

  • Success — A successful operation.
  • SuccessWithWarning — A successful operation; however, there are messages returned in the response that you should examine.
  • Failure — The operation failed; the response also contains one or more error messages explaining the failure.
  • FailureWithWarning — The operation failed and there are messages returned in the response that you should examine.

And, we have completed the PayPal transaction flow for Express Checkout. These are just the basics and might miss a few stuff. I suggest you go through the following links too for a better understanding of everything:

For Reference:
  1. PayPal Name-Value Pair API Basics – PayPal Developer
  2. How to Create One-Time Payments Using Express Checkout – PayPal Developer

Building interactive elements with HTML and javascript: Interact.js + resizing

{ Repost from my personal blog @ https://blog.codezero.xyz/building-interactive-elements-with-html-and-javascript-interact-js-resizing/ }

In a few of the past blog posts, we saw about implementing resizing with HTML and javascript. The functionality was pretty basic with simple resizing. In the last blog post we saw about interact.js.

interact.js is a lightweight, standalone JavaScript module for handling single-pointer and multi-touch drags and gestures with powerful features including inertia and snapping.

Getting started with Interact.js

You have multiple option to include the library in your project.

  • You can use bower to install (bower install interact) (or)
  • npm (npm install interact.js) (or)
  • You could directly include the library from a CDN (https://cdnjs.cloudflare.com/ajax/libs/interact.js/1.2.6/interact.min.js).
Implementing resizing

Let’s create a simple box using HTML. We’ll add a class called resizable to it so that we can reference it to initialize Interact.js

<div class="resizable">  
    Use right/bottom edge to resize
</div>

We need to create an interact instance. Once the instance is created, we have to call the resizable method on it to add resize support to the div.

interact('.resizable')
  .resizable({
    edges: { right: true, bottom: true }
  })
  .on('resizemove', function (event) {
    

  });

Inside the resizable method, we can pass configuration options. The edgesconfig key allows us to specify on which all edges, resizing should be allowed. Right now, we have allowed on the right and bottom edges. Similarly we can have resizing support in the top and left edges too.

The resizemove event is triggered by interact every time the user tries to resize the div. From the event, we can get the box that is being resized, (i.e) the target by accessing event.target.

The event object also provides us event.rect.width and event.rect.height which is the width and height of the div after resizing. We’ll not set this as the width of the div so that, the user is able to see the width change.

var target = event.target;
    // update the element's style
    target.style.width  = event.rect.width + 'px';
    target.style.height = event.rect.height + 'px';

We can also instruct Interact.js to preserve the aspect ratio of the box by adding an option preserveAspectRatio: true to the configuration object passed to resizable method during initialization.

JavaScript
interact('.resizable')
  .resizable({
    edges: { right: true, bottom: true }
  })
  .on('resizemove', function (event) {
    var target = event.target;

    // update the element's style
    target.style.width  = event.rect.width + 'px';
    target.style.height = event.rect.height + 'px';
  });

Resizing and drag-drop (with Interact.js) were used to create the Scheduler tool at Open Event. The tool allows event/track organizers to easily arrange the sessions into their respective rooms by drag-drop and also to easily change the timings of the events by resizing the event block. The entire source code of the scheduler can be viewed at app/static/js/admin/event/scheduler.js in the Open Event Organizer server’s GitHub repository.

Demo:
https://jsfiddle.net/xdfocdty/

Implementing revisioning feature in Open Event

{ Repost from my personal blog @ https://blog.codezero.xyz/implementing-revisioning-feature-in-open-event }

As I said in my previous blog post about Adding revisioning to SQLAlchemy Models,

In an application like Open Event, where a single piece of information can be edited by multiple users, it’s always good to know who changed what. One should also be able to revert to a previous version if needed.

Let’s have a quick run through on how we can enable SQLAlchemy-Continuum on our project.

  1. Install the library SQLAlchemy-Continuum with pip
  2. Add __versioned__ = {} to all the models that need to be versioned.
  3. Call make_versioned() before the models are defined
  4. Call configure_mappers from SQLAlchemy after declaring all the models.

Example:

import sqlalchemy as sa  
from sqlalchemy_continuum import make_versioned

# Must be called before defining all the models
make_versioned()

class Event(Base):

    __tablename__ = 'events'
    __versioned__ = {}  # Must be added to all models that are to be versioned

    id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
    name = sa.Column(sa.String)
    start_time = sa.Column(db.DateTime, nullable=False)
    end_time = sa.Column(db.DateTime, nullable=False)
    description = db.Column(db.Text)
    schedule_published_on = db.Column(db.DateTime)

# Must be called after defining all the models
sa.orm.configure_mappers()

We have SQLAlchemy-Continuum enabled now. You can do all the read/write operations as usual. (No change there).

Now, for the part where we give the users an option to view/restore revisions. The inspiration for this, comes from wordpress’s wonderful revisioning functionality.

The layout is well designed. The differences are shown in an easy-to-read form. The slider on top makes it intuitive to move b/w revisions. We have a Restore this Revision button on the top-right to switch to that revision.

A similar layout is what we would like to achieve in Open Event.

  1. A slider to switch b/w sessions
  2. A pop-over infobox on the slider to show who made that change
  3. A button to switch to that selected revision.
  4. The colored-differences shown in side-by-side manner.

To make all this a bit easier, SQLAlchemy-Continuum provides us with some nifty methods.

count_versions is a method that allows us to know the number of revisions a record has.

event = session.query(Event).get(1)  
count = count_versions(event)  # number of versions of that event

Next one is pretty cool. All the version objects have a property called as changeset which holds a dict of changed fields in that version.

event = Event(name=u'FOSSASIA 2016', description=u'FOSS Conference in Asia')  
session.add(article)  
session.commit(article)

version = event.versions[0]  # first version  
version.changeset  
# {
#   'id': [None, 1],
#   'name': [None, u'FOSSASIA 2016'],
#   'description': [None, u'FOSS Conference in Asia']
# }

event.name = u'FOSSASIA 2017'  
session.commit()

version = article.versions[1]  # second version  
version.changeset  
# {
#   'name': [u'FOSSASIA 2016', u'FOSSASIA 2017'],
# }

As you can see, dict holds the fields that changed and the content the changed (before and after). And this is what we’ll be using for generating those pretty diffs that the guys and girls over at wordpress.com have done. And for this we’ll be using two things.

  1. A library named diff-match-patch. It is a library from Google which offers robust algorithms to perform the operations required for synchronizing plain text.
  2. A small recipe from from code.activestate.com Line-based diffs with the necessary HTML markup for styling insertions and deletions.
import itertools  
import re

import diff_match_patch

def side_by_side_diff(old_text, new_text):  
    """
    Calculates a side-by-side line-based difference view.

    Wraps insertions in <ins></ins> and deletions in <del></del>.
    """
    def yield_open_entry(open_entry):
        """ Yield all open changes. """
        ls, rs = open_entry
        # Get unchanged parts onto the right line
        if ls[0] == rs[0]:
            yield (False, ls[0], rs[0])
            for l, r in itertools.izip_longest(ls[1:], rs[1:]):
                yield (True, l, r)
        elif ls[-1] == rs[-1]:
            for l, r in itertools.izip_longest(ls[:-1], rs[:-1]):
                yield (l != r, l, r)
            yield (False, ls[-1], rs[-1])
        else:
            for l, r in itertools.izip_longest(ls, rs):
                yield (True, l, r)

    line_split = re.compile(r'(?:r?n)')
    dmp = diff_match_patch.diff_match_patch()

    diff = dmp.diff_main(old_text, new_text)
    dmp.diff_cleanupSemantic(diff)

    open_entry = ([None], [None])
    for change_type, entry in diff:
        assert change_type in [-1, 0, 1]

        entry = (entry.replace('&', '&amp;')
                      .replace('<', '&lt;')
                      .replace('>', '&gt;'))

        lines = line_split.split(entry)

        # Merge with previous entry if still open
        ls, rs = open_entry

        line = lines[0]
        if line:
            if change_type == 0:
                ls[-1] = ls[-1] or ''
                rs[-1] = rs[-1] or ''
                ls[-1] = ls[-1] + line
                rs[-1] = rs[-1] + line
            elif change_type == 1:
                rs[-1] = rs[-1] or ''
                rs[-1] += '<ins>%s</ins>' % line if line else ''
            elif change_type == -1:
                ls[-1] = ls[-1] or ''
                ls[-1] += '<del>%s</del>' % line if line else ''

        lines = lines[1:]

        if lines:
            if change_type == 0:
                # Push out open entry
                for entry in yield_open_entry(open_entry):
                    yield entry

                # Directly push out lines until last
                for line in lines[:-1]:
                    yield (False, line, line)

                # Keep last line open
                open_entry = ([lines[-1]], [lines[-1]])
            elif change_type == 1:
                ls, rs = open_entry

                for line in lines:
                    rs.append('<ins>%s</ins>' % line if line else '')

                open_entry = (ls, rs)
            elif change_type == -1:
                ls, rs = open_entry

                for line in lines:
                    ls.append('<del>%s</del>' % line if line else '')

                open_entry = (ls, rs)

    # Push out open entry
    for entry in yield_open_entry(open_entry):
        yield entry

So, what we have to do is,

  1. Get the changeset from a version
  2. Run each field’s array containing the old and new text through the side_by_side_diff method.
  3. Display the output on screen.
  4. Use the markups <ins/> and <del/> to style changes.

So, we do the same for each version by looping through the versions array accessible from an event record.

For the slider, noUiSlider javascript library was used. Implementation is simple.

<div id="slider"></div>

<script type="text/javascript">  
    $(function () {
        var slider = document.getElementById('slider');
        noUiSlider.create(slider, {
            start: [0],
            step: 1,
            range: {
                'min': 0,
                'max': 5
            }
        });
    });
</script>

This would create a slider that can go from 0 to 5 and will start at position 0.

By listening to the update event of the slider, we’re able to change which revision is displayed.

slider.noUiSlider.on('update', function (values, handle) {  
    var value = Math.round(values[handle]);
    // the current position of the slider
    // do what you have to do to change the displayed revision
});

And to get the user who caused a revision, you have to access the user_idparameter of the transaction record of a particular version.

event = session.query(Event).get(1)  
version_one = event.versions[0]  
transaction = transaction_class(version_one)  
user_id = transaction.user_id

So, with the user ID, you can query the user database to get the user who made that revision.

The user_id is automatically populated if you’re using Flask, Flask-login and SQLAlchemy-Continuum’s Flask Plugin. Enabling the plugin is easy.

from sqlalchemy_continuum.plugins import FlaskPlugin  
from sqlalchemy_continuum import make_versioned

make_versioned(plugins=[FlaskPlugin()])

This is not a very detailed blog post. If you would like to see the actual implementation, you can checkout the Open Event repository over at GitHub. Specifically, the file browse_revisions.html.

The result is,

Still needs some refinements in the UI. But, it gets the job done :wink:

Adding revisioning to SQLAlchemy Models

{ Repost from my personal blog @ https://blog.codezero.xyz/adding-revisioning-to-sqlalchemy-models }

In an application like Open Event, where a single piece of information can be edited by multiple users, it’s always good to know who changed what. One should also be able to revert to a previous version if needed. That is where revisioning comes into picture.

We use SQLAlchemy as the database toolkit and ORM. So, we wanted a versioning tool that would work well with our existing setup. That’s when I came across SQLAlchemy-Continuum – a versioning extension for SQLAlchemy.

Installation

The installation of the module is just like any other python library. (don’t forget to add it to your requirements.txt file, if you have one)

pip install SQLAlchemy-Continuum
Setup

Now, it’s time to enable SQLAlchemy-Continuum for the required models.

Let’s consider an Event model.

import sqlalchemy as sa

class Event(Base):
    __tablename__ = 'events'
    id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
    name = sa.Column(sa.String)
	start_time = sa.Column(db.DateTime, nullable=False)
    end_time = sa.Column(db.DateTime, nullable=False)
    description = db.Column(db.Text)
    schedule_published_on = db.Column(db.DateTime)

We need to do three things to enable SQLAlchemy-Continuum.

  1. Call make_versioned() before the model(s) is/are defined.
  2. Add __versioned__ = {} to all the models that we want to be versioned
  3. Call configure_mappers from SQLAlchemy after declaring all the models.
import sqlalchemy as sa
from sqlalchemy_continuum import make_versioned

# Must be called before defining all the models
make_versioned()

class Event(Base):

    __tablename__ = 'events'
    __versioned__ = {}  # Must be added to all models that are to be versioned

    id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
    name = sa.Column(sa.String)
	start_time = sa.Column(db.DateTime, nullable=False)
    end_time = sa.Column(db.DateTime, nullable=False)
    description = db.Column(db.Text)
    schedule_published_on = db.Column(db.DateTime)

# Must be called after defining all the models
sa.orm.configure_mappers()

SQLAlchemy-Continuum creates two tables:

  1. events_version which stores the version history for the Event model linked to the transaction table via a foreign key
  2. transaction which stores information about each transaction (like the user who performed the transaction, transaction datetime, etc)

SQLAlchemy-Continuum also adds listeners to Event to record all create, update, delete actions.

Usage

All the CRUD (Create, read, update, delete) operations can be done as usual. SQLAlchemy-Continuum takes care of creating a version record for each CUD operation. The versions can be easily accessed using the versions property that is now available in the Event model.

event = Event(name="FOSSASIA 2017", description="Open source conference in asia")
session.add(event)  # Event added to transaction
session.commit() # Transaction comitted and event recored created

# This would have created the first version record which can be accessed
event.versions[0].name == "FOSSASIA 2017"

# Lets make some changes to the recored.
event.name = "FOSSASIA 2016"
session.add(event)
session.commit()

# This would have created the second version record which can be accessed
event.versions[1].name == "FOSSASIA 2016"

# The first version record still remains and can be accessed
event.versions[0].name == "FOSSASIA 2017"

So, that’s how basic versioning can be implemented in SQLAlchemy using SQLAlchemy-Continuum.

Open Event: Planning the Scheduler UI

{ Repost from my personal bloghttps://blog.codezero.xyz/planning-the-scheduler-ui/ }

In the first phase of the GSoC coding period, Saptak Sengupta and myself have been working on the Scheduler UI.

The Scheduler UI would allow the organizers to graphically schedule/manage the sessions in their conference/event.

Event-Organizer by Josh Greco has been a major inspiration for the timeline design. The css styles have been borrowed from that project.

After some research, we have decided to use the interact.js javascript library for implementing drag-drop and resizing functionality to the project and lodash for array/collection manipulations and for some useful utility functions.

The following tasks have been formulated:

  • Drag-and-drop interface to add session blocks into a timeline
  • Resize sessions to change time
  • load live data from the server using the API
  • Handle session clashes within a track properly
  • Add Track button that opens up a modal to add a track.
  • Search option for unscheduled sessions
  • Save each session change/update using the API
  • Option to Export timeline as pdf
  • Option to export timeline/calendar as iCal

An umbrella issue (#349) has been created in the open-event-orga-serverrespository to track the progress. There will be a separate issue created for each task when that task is being worked upon and the same would be referred to in the umbrella issue.

We are targeting GSOC 2016: Milestone 4 (Due by June 12, 2016) to finish the Scheduler UI and we are confident that we’ll be able to reach our target. *fingers-crossed* :sweat_smile:

To know more about the Scheduler UI, read Enhancing the Open Event Server: Scheduler UI.

Enhancing the Open Event Server: Scheduler UI

{ Repost from my personal blog @ https://blog.codezero.xyz/enhancing-open-event-scheduler-one/ }

The community bonding period went pretty well. Worked on the OpenTechSummit 2016‘s website. Fixed a few issues. Got to know the team. :smile:

Now comes the coding period which started yesterday (23rd May 2016).
In the first phase, I (Niranjan Rajendran) will be primarily working on creating a Scheduler tool for each event. Saptak Sengupta and myself will be working on this.

The Scheduler would allow the organizers to graphically schedule/manage the sessions in their conference/event. The proposed features for the Scheduler are:

  1. Drag-and-drop sessions into tracks on a timeline.
  2. Change the timings of the sessions by resizing the sessions’ element on the timeline.
  3. Create a new event by dragging a time period on the timeline.
  4. Create new tracks.
  5. Switch between timelines of the different days of the event.
  6. Load existing sessions from the database into the timeline.
  7. Print the timeline or convert it into a PDF file.
  8. Export the schedule in different formats such as iCalendar.

The scheduler UI would not be practical on small screens (Mobile phones) and would be hard to use. So, we will be thinking of some alternative – a simpler interface for smaller screens.