Adding a Last Modified At column in Open Event Server

This blog article will illustrate how, with the help of SQLAlchemy, a last modified at column, with complete functionality can be added to the Open Event Server database. To illustrate the process, the blog article will discuss adding the column to the sessions api. Since last modified at is a time field, and will need to be updated each time user successfully updates the session, the logic to implement will be a slightly more complex than a mere addition of a column to the table.

The first obvious step will comprise of adding the column to the database table. To achieve the same, the column will have to be added to the model for the sessions table, as well as the schema.

In app/api/schema/sessions.py:

...
class SessionSchema(Schema):
   """
   Api schema for Session Model
   """
   ...
   last_modified_at = fields.DateTime(dump_only=True)
   ...

And in app/models/sessions.py:

import pytz
...

class Session(db.Model):
   """Session model class"""
   __tablename__ = 'sessions'
   __versioned__ = {
       'exclude': []
   }
   ...
   last_modified_at = db.Column(db.DateTime(timezone=True),   
   default=datetime.datetime.utcnow)
   def init(self, ..., last_modified_at=None))
     #inside init method
     ...
     self.last_modified_at = datetime.datetime.now(pytz.utc)
     ...

NOTE: The users for the open event organiser server will be operating in multiple time zones and hence it is important for all the times to be in sync, hence the open event database maintains all the time in UTC timezone (python’s pytz module takes care of converting user’s local time into UTC time while storing, thus unifying the timezones.) From this, it directly follows that the time needs to be timezone aware hence timezone=true is passed, while defining the column.

Next, while initialising an object of this class, the last modified time is the time of creation, and hence

datetime.now(pytz.utc) is set as the initial value which basically stores the current time in UTC timezone format.

Finally, the logic for updating the last modified at column every time any other value changes for a session record needs to be implemented. SQLAlchemy provides an inbuilt support for detecting update and insert events which have been used to achieve the goal. To quote the official SQLAlchemy Docs,  “SQLAlchemy includes an event API which publishes a wide variety of hooks into the internals of both SQLAlchemy Core and ORM.

@event.listens_for(Session, 'after_update')
def receive_after_update(mapper, connection, target):
  target.last_modified_at = datetime.datetime.now(pytz.utc)

The listens_for() decorator is used to register the event according to the arguments passed to it. In our case, it will register any event on the Session API (sessions table), whenever it updates.

The corresponding function defined below the decorator, receive_after_update(mapper, connection, target) is then called, and session model (table) is the the registered target with the event. It sets the value of the last_modified_at to the current time in the UTC timezone as expected.

Lastly, since the changes have been made to the database schema, the migration file needs to be generated, and the database will be upgraded to alter the structure.

The sequence of steps to be followed on the CLI will be

> python manage.py db migrate
> python manage.py db upgrade

Resources

UI automated testing using Selenium in Badgeyay

With all the major functionalities packed into the badgeyay web application, it was time to add some automation testing to automate the review process in case of known errors and check if code contribution by contributors is not breaking anything. We decided to go with Selenium for our testing requirements.

What is Selenium?

Selenium is a portable software-testing framework for web applications. Selenium provides a playback (formerly also recording) tool for authoring tests without the need to learn a test scripting language. In other words, Selenium does browser automation:, Selenium tells a browser to click some element, populate and submit a form, navigate to a page and any other form of user interaction.

Selenium supports multiple languages including C#, Groovy, Java, Perl, PHP, Python, Ruby and Scala. Here, we are going to use Python (and specifically python 2.7).

First things first:
To install these package run this code on the CLI:

pip install selenium==2.40
pip install nose

Don’t forget to add them in the requirements.txt file

Web Browser:
We also need to have Firefox installed on your machine.

Writing the Test
An automated test automates what you’d do via manual testing – but it is done by the computer. This frees up time and allows you to do other things, as well as repeat your testing. The test code is going to run a series of instructions to interact with a web browser – mimicking how an actual end user would interact with an application. The script is going to navigate the browser, click a button, enter some text input, click a radio button, select a drop down, drag and drop, etc. In short, the code tests the functionality of the web application.

A test for the web page title:

import unittest
from selenium import webdriver

class SampleTest(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        cls.driver = webdriver.Firefox()
        cls.driver.get('http://badgeyay-dev.herokuapp.com/')

    def test_title(self):
        self.assertEqual(self.driver.title, 'Badgeyay')

    @classmethod
    def tearDownClass(cls):
        cls.driver.quit()

 

Run the test using nose test.py

Clicking the element
For our next test, we click the menu button, and check if the menu becomes visible.

elem = self.driver.find_element_by_css_selector(".custom-menu-content")
self.driver.find_element_by_css_selector(".glyphicon-th").click()
self.assertTrue(elem.is_displayed())

 

Uploading a CSV file:
For our next test, we upload a CSV file and see if a success message pops up.

def test_upload(self):
        Imagepath = os.path.abspath(os.path.join(os.getcwd(), 'badges/badge_1.png'))
        CSVpath = os.path.abspath(os.path.join(os.getcwd(), 'sample/vip.png.csv'))
        self.driver.find_element_by_name("file").send_keys(CSVpath)
        self.driver.find_element_by_name("image").send_keys(Imagepath)
        self.driver.find_element_by_css_selector("form .btn-primary").click()
        time.sleep(3)
        success = self.driver.find_element_by_css_selector(".flash-success")
        self.assertIn(u'Your badges has been successfully generated!', success.text)

 

The entire code can be found on: https://github.com/fossasia/badgeyay/tree/development/app/tests

We can also use the Phantom.js package along with Selenium for UI testing purposes without opening a web browser. We use this for badgeyay to run the tests for every commit in Travis CI which cannot open a program window.

Resources

Make Flask Fast and Reliable – Simple Steps

Flask is a microframework for Python, which is mostly used in web-backend development.There are projects in FOSSASIA that are using flask for development purposes such as Open Event Server, Query Server, Badgeyay. Optimization is indeed one of the most important steps for a successful software product. So, in this post some few off- the-hook tricks will be shown which will make your flask-app more fast and reliable.

Flask-Compress

  1. Flask-Compress is a python package which basically provides de-facto lossless compression  to your Flask application.
  2. Enough with the theory, now let’s understand the coding part:
    1. First install the module

2. Then for a basic setup

3.That’s it! All it takes is just few lines of code to make your flask app optimized .To know more about the module check out flask-compress module.

Requirements Directory

  1. A common practice amongst different FOSSASIA  projects which involves dividing requirements.txt files for development,testing as well as production.
  2. Basically when projects either use TRAVIS CI for testing or are deployed to Cloud Services like Heroku, there are some modules which are not really required at some places.  For example: gunicorn is only required for deployment purposes and not for development.
  3. So how about we have a separate directory wherein different .txt files are created for different purposes.
  4. Below is the image of file directory structure followed for requirements in badgeyay project.

  1. As you can see different .txt files are created for different purposes
    1. dev.txt – for development
    2. prod.txt – for production(i.e. deployment)
    3. test.txt – for testing.

Resources

Open Event API Server: Implementing FAQ Types

In the Open Event Server, there was a long standing request of the users to enable the event organisers to create a FAQ section.

The API of the FAQ section was implemented subsequently. The FAQ API allowed the user to specify the following request schema

{
 "data": {
   "type": "faq",
   "relationships": {
     "event": {
       "data": {
         "type": "event",
         "id": "1"
       }
     }
   },
   "attributes": {
     "question": "Sample Question",
     "answer": "Sample Answer"
   }
 }
}

 

But, what if the user wanted to group certain questions under a specific category. There was no solution in the FAQ API for that. So a new API, FAQ-Types was created.

Why make a separate API for it?

Another question that arose while designing the FAQ-Types API was whether it was necessary to add a separate API for it or not. Consider that a type attribute was simply added to the FAQ API itself. It would mean the client would have to specify the type of the FAQ record every time a new record is being created for the same. This would mean trusting that the user will always enter the same spelling for questions falling under the same type. The user cannot be trusted on this front. Thus the separate API made sure that the types remain controlled and multiple entries for the same type are not there.

Helps in handling large number of records:

Another concern was what if there were a large number of FAQ records under the same FAQ-Type. Entering the type for each of those questions would be cumbersome for the user. The FAQ-Type would also overcome this problem

Following is the request schema for the FAQ-Types API

{
 "data": {
   "attributes": {
     "name": "abc"
   },
   "type": "faq-type",
   "relationships": {
     "event": {
       "data": {
         "id": "1",
         "type": "event"
       }
     }
   }
 }
}

 

Additionally:

  • FAQ to FAQ-type is a many to one relation.
  • A single FAQ can only belong to one Type
  • The FAQ-type relationship will be optional, if the user wants different sections, he/she can add it ,if not, it’s the user’s choice.

Related links

Implementing Permissions for Orders API in Open Event API Server

Open Event API Server Orders API is one of the core APIs. The permissions in Orders API are robust and secure enough to ensure no leak on payment and ticketing.The permission manager provides the permissions framework to implement the permissions and proper access controls based on the dev handbook.

The following table is the permissions in the developer handbook.

 

List View Create Update Delete
Superadmin/admin
Event organizer [1] [1] [1] [1][2] [1][3]
Registered user [4]
Everyone else
  1. Only self-owned events
  2. Can only change order status
  3. A refund will also be initiated if paid ticket
  4. Only if order placed by self

Super Admins and admins are allowed to create any order with any amount but any coupon they apply is not consumed on creating order. They can update almost every field of the order and can provide any custom status to the order. Permissions are applied with the help of Permission Manager which takes care the authorization roles. For example, if a permission is set based on admin access then it is automatically set for super admin as well i.e., to the people with higher rank.

Self-owned events

This allows the event admins, Organizer and Co-Organizer to manage the orders of the event they own. This allows then to view all orders and create orders with or without discount coupon with any custom price and update status of orders. Event admins can provide specific status while others cannot

if not has_access('is_coorganizer', event_id=data['event']):
   data['status'] = 'pending'

And Listing requires Co-Organizer access

elif not has_access('is_coorganizer', event_id=kwargs['event_id']):
   raise ForbiddenException({'source': ''}, "Co-Organizer Access Required")

Can only change order status

The organizer cannot change the order fields except the status of the order. Only Server Admin and Super Admins are allowed to update any field of the order.

if not has_access('is_admin'):
   for element in data:
       if element != 'status':
           setattr(data, element, getattr(order, element))

And Delete access is prohibited to event admins thus only Server admins can delete orders by providing a cancelling note which will be provided to the Attendee/Buyer.

def before_delete_object(self, order, view_kwargs):
   if not has_access('is_coorganizer', event_id=order.event.id):
       raise ForbiddenException({'source': ''}, 'Access Forbidden')

Registered User

A registered user can create order with basic details like the attendees’ records and payment method with fields like country and city. They are not allowed to provide any custom status to the order they are creating. All orders will be set by default to “pending”

Also, they are not allowed to update any field in their order. Any status update will be done internally thus maintaining the security of Order System. Although they are allowed to view their place orders. This is done by comparing their logged in user id with the user id of the purchaser.

if not has_access('is_coorganizer_or_user_itself', event_id=order.event_id, user_id=order.user_id):
   return ForbiddenException({'source': ''}, 'Access Forbidden')

Event Admins

The event admins have one more restriction, as an event admin, you cannot provide discount coupon and even if you do it will be ignored.

# Apply discount only if the user is not event admin
if data.get('discount') and not has_access('is_coorganizer', event_id=data['event']):

Also an event admin any amount you will provide on creating order will be final and there will be no further calculation of the amount will take place

if not has_access('is_coorganizer', event_id=data['event']):
   TicketingManager.calculate_update_amount(order)

Creating Attendees Records

Before sending a request to Orders API it is required to create to attendees mapped to some ticket and for this registered users are allowed to create the attendees without adding a relationship of the order. The mapping with the order is done internally by Orders API and its helpers.

Resources

  1. Dev Handbook – Niranjan R
    The Open Event Developer Handbook
  2. Flask-REST-JSONAPI Docs
    Permissions and Data layer | Flask-REST-JSONAPI
  3. A guide to use permission manager in API Server
    http://blog.fossasia.org/a-guide-to-use-permission-manager-in-open-event-api-server/

 

Generating Ticket PDFs in Open Event API Server

In the ordering system of Open Event API Server, there is a requirement to send email notifications to the attendees. These attendees receive the URL of the pdf of the generated ticket. On creating the order, first the pdfs are generated and stored in the preferred storage location and then these are sent to the users through the email.

Generating PDF is a simple process, using xhtml2pdf we can generate PDFs from the html. The generated pdf is then passed to storage helpers to store it in the desired location and pdf-url is updated in the attendees record.

Sample PDF

PDF Template

The templates are written in HTML which is then converted using the module xhtml2pdf.
To store the templates a new directory was created at  app/templates where all HTML files are stored. Now, The template directory needs to be updated at flask initializing app so that template engine can pick the templates from there. So in app/__init__.py we updated flask initialization with

template_dir = os.path.dirname(__file__) + "/templates"

app = Flask(__name__, static_folder=static_dir, template_folder=template_dir)

This allows the template engine to pick the templates files from this template directory.

Generating PDFs

Generating PDF is done by rendering the html template first. This html content is then parsed into the pdf

file = open(dest, "wb")

pisa.CreatePDF(cStringIO.StringIO(pdf_data.encode('utf-8')), file)

file.close()

The generated pdf is stored in the temporary location and then passed to storage helper to upload it.

uploaded_file = UploadedFile(dest, filename)

upload_path = UPLOAD_PATHS['pdf']['ticket_attendee'].format(identifier=get_file_name())

new_file = upload(uploaded_file, upload_path)

This generated pdf path is returned here

Rendering HTML and storing PDF

for holder in order.ticket_holders:

  if holder.id != current_user.id:

      pdf = create_save_pdf(render_template('/pdf/ticket_attendee.html', order=order, holder=holder))

  else:

      pdf = create_save_pdf(render_template('/pdf/ticket_purchaser.html', order=order))

  holder.pdf_url = pdf

  save_to_db(holder)

The html is rendered using flask template engine and passed to create_save_pdf and link is updated on the attendee record.

Sending PDF on email

These pdfs are sent as a link to the email after creating the order. Thus a ticket is sent to each attendee and a summarized order details with attendees to the purchased.

send_email(

  to=holder.email,

  action=TICKET_PURCHASED_ATTENDEE,

  subject=MAILS[TICKET_PURCHASED_ATTENDEE]['subject'].format(

      event_name=order.event.name,

      invoice_id=order.invoice_number

  ),

  html= MAILS[TICKET_PURCHASED_ATTENDEE]['message'].format(

      pdf_url=holder.pdf_url,

      event_name=order.event.name

  )

)

References

  1. Readme – xhtml2pdf
    https://github.com/xhtml2pdf/xhtml2pdf/blob/master/README.rst
  2. Using xhtml2pdf and create pdfs
    https://micropyramid.com/blog/generating-pdf-files-in-python-using-xhtml2pdf/

 

Discount Codes in Open Event Server

The Open Event System allows usage of discount codes with tickets and events. This blogpost describes what types of discount codes are present and what endpoints can be used to fetch and update details.

In Open Event API Server, each event can have two types of discount codes. One is ‘event’ discount code, while the other is ‘ticket’ discount code. As the name suggests, the event discount code is an event level discount code and the ticket discount code is ticket level.

Now each event can have only one ‘event’ discount code and is accessible only to the server admin. The Open Event server admin can create, view and update the ‘event’ discount code for an event. The event discount code followsDiscountCodeEvent Schema. This schema is inherited from the parent class DiscountCodeSchemaPublic. To save the unique discount code associated with an event, the event model’s discount_code_id field is used.

The ‘ticket’ discount is accessible by the event organizer and co-organizer. Each event can have any number of ‘ticket’ discount codes. This follows the DiscountCodeTicket schema, which is also inherited from the same base class ofDiscountCodeSchemaPublic. The use of the schema is decided based on the value of the field ‘used_for’ which can have the value either ‘events’ or ‘tickets’. Both the schemas have different relationships with events and marketer respectively.

We have the following endpoints for Discount Code events and tickets:
‘/events/<int:event_id>/discount-code’
‘/events/<int:event_id>/discount-codes’

The first endpoint is based on the DiscountCodeDetail class. It returns the detail of one discount code which in this case is the event discount code associated with the event.

The second endpoint is based on the DiscountCodeList class which returns a list of discount codes associated with an event. Note that this list also includes the ‘event’ discount code, apart from all the ticket discount codes.

class DiscountCodeFactory(factory.alchemy.SQLAlchemyModelFactory):
   class Meta:
       model = DiscountCode
       sqlalchemy_session = db.session
event_id = None
user = factory.RelatedFactory(UserFactory)
user_id = 1


Since each discount code belongs to an event(either directly or through the ticket), the factory for this has event as related factory, but to check for 
/events/<int:event_id>/discount-code endpoint we first need the event and then pass the discount code id to be 1 for dredd to check this. Hence, event is not included as a related factory, but added as a different object every time a discount code object is to be used.

@hooks.before("Discount Codes > Get Discount Code Detail of an Event > Get Discount Code Detail of an Event")
def event_discount_code_get_detail(transaction):
   """
   GET /events/1/discount-code
   :param transaction:
   :return:
   """
   with stash['app'].app_context():
       discount_code = DiscountCodeFactory()
       db.session.add(discount_code)
       db.session.commit()
       event = EventFactoryBasic(discount_code_id=1)
       db.session.add(event)
       db.session.commit()


The other tests and extended documentation can be found 
here.

References:

Open Event Server: Getting The Identity From The Expired JWT Token In Flask-JWT

The Open Event Server uses JWT based authentication, where JWT stands for JSON Web Token. JSON Web Tokens are an open industry standard RFC 7519 method for representing claims securely between two parties. [source: https://jwt.io/]

Flask-JWT is being used for the JWT-based authentication in the project. Flask-JWT makes it easy to use JWT based authentication in flask, while on its core it still used PyJWT.

To get the identity when a JWT token is present in the request’s Authentication header , the current_identity proxy of Flask-JWT can be used as follows:

@app.route('/example')
@jwt_required()
def example():
   return '%s' % current_identity

 

Note that it will only be set in the context of function decorated by jwt_required(). The problem with the current_identity proxy when using jwt_required is that the token has to be active, the identity of an expired token cannot be fetched by this function.

So why not write a function on our own to do the same. A JWT token is divided into three segments. JSON Web Tokens consist of three parts separated by dots (.), which are:

  • Header
  • Payload
  • Signature

The first step would be to get the payload, that can be done as follows:

token_second_segment = _default_request_handler().split('.')[1]

 

The payload obtained above would still be in form of JSON, it can be converted into a dict as follows:

payload = json.loads(token_second_segment.decode('base64'))

 

The identity can now be found in the payload as payload[‘identity’]. We can get the actual user from the paylaod as follows:

def jwt_identity(payload):
   """
   Jwt helper function
   :param payload:
   :return:
   """
   return User.query.get(payload['identity'])

 

Our final function will now be something like:

def get_identity():
   """
   To be used only if identity for expired tokens is required, otherwise use current_identity from flask_jwt
   :return:
   """
   token_second_segment = _default_request_handler().split('.')[1]
   missing_padding = len(token_second_segment) % 4
   payload = json.loads(token_second_segment.decode('base64'))
   user = jwt_identity(payload)
   return user

 

But after using this function for sometime, you will notice that for certain tokens, the system will raise an error saying that the JWT token is missing padding. The JWT payload is base64 encoded, and it requires the payload string to be a multiple of four. If the string is not a multiple of four, the remaining spaces can pe padded with extra =(equal to) signs. And since Python 2.7’s .decode doesn’t do that by default, we can accomplish that as follows:

missing_padding = len(token_second_segment) % 4

# ensures the string is correctly padded to be a multiple of 4
if missing_padding != 0:
   token_second_segment += b'=' * (4 - missing_padding)

 

Related links:

Stripe Authorization In Open Event API Server

The Open Event System supports payments through stripe. Stripe is a suite of APIs that powers commerce for businesses of all sizes. This blogpost covers testing of Stripe Authorization Schema and endpoints in the API Server.

The Stripe Authorization class provides the following endpoints:

'/stripe-authorization'
'/stripe-authorization/<int:id>'
'/events/<int:event_id>/stripe-authorization'
'/events/<event_identifier>/stripe-authorization'


In the pull request made for adding documentation and tests, these two endpoints were removed:

'stripe_authorization_list',
'/events/<int:event_id>/stripe-authorization',
'/events/<event_identifier>/stripe-authorization'

This is because each event can have only one stripe authorization, so there can not exist a list of stripe authorization objects related to an event.

The ‘stripe_authorization_list’ endpoint is made POST only. This is because Open Event does not allow individual resources’ list to be accessible. Since, there is no endpoint which returns a list of Stripe Authorizations the StripeAuthorizationList(ResourceListis removed.

The ResourceDetail class was modified to add a query to support  results from ‘/events/<int:event_id>/stripe-authorization’ endpoint suThe view_kwargs for the detail endpoint has to contain the resource id, so event_id from view_kwags is used to get the id for stripe authorization.

stripe_authorization = self.session.query(StripeAuthorization).filter_by(event_id=view_kwargs['event_id']).one()
view_kwargs['id'] = stripe_authorization.id

Writing Test for Documentation

(Tests for the /events/1/stripe-authorization  is described here, for others please refer to links in additional references.)

To test the  /events/1/stripe-authorization endpoint for GET, we first insert a Stripe Authorization object into the database which will then be retrieved by the GET request and then compared with the expected response body from the documentation file.

Since stripe-auth has a required relationship with event class, an event must also exist for strie auth object to be created. The event is also required because the endpoint ‘events/’ expects an event object to exist. The StripeAuthorizationFactory takes care of this with event as a RelatedFactory. So when a StripeAuthorization object is inserted, an event is created first and passed as the required relationship to stripe_list_post endpoint.

The event is related to the stripe object by setting event_id = 1 in the factory.

Adding the pre-test hook for GET:

@hooks.before("StripeAuthorization > Stripe Authorization for an Event > Get Stripe Authorization Details of an Event")
def event_stripe_authorization_get_detail(transaction):
   """
   GET /events/1/stripe-authorization
   :param transaction:
   :return:
   """
   with stash['app'].app_context():
       stripe = StripeAuthorizationFactory()
       db.session.add(stripe)
       db.session.commit()


The expected response for this request can be found
 here.

Additional References:

Copying Event in Open Event API Server

The Event Copy feature of Open Event API Server provides the ability to create a xerox copy of event copies with just one API call. This feature creates the complete copy of event by copying the related objects as well like tracks, sponsors, micro-locations, etc. This API is based on the simple method where an object is first removed is from current DB session and then applied make_transient. Next step is to remove the unique identifying columns like “id”, “identifier” and generating the new identifier and saving the new record. The process seems simple but becomes a little complex when you have to generate copies of media files associated and copies of related multiple objects ensuring no orders, attendees, access_codes relations are copied.

Initial Step

The first thing to copy the event is first to get the event object and all related objects first

if view_kwargs.get('identifier').isdigit():
   identifier = 'id'

event = safe_query(db, Event, identifier, view_kwargs['identifier'], 'event_'+identifier)

Next thing is to get all related objects to this event.

Creating the new event

After removing the current event object from “db.session”, It is required to remove “id” attribute and regenerate “identifier” of the event.

db.session.expunge(event)  # expunge the object from session
make_transient(event)
delattr(event, 'id')
event.identifier = get_new_event_identifier()
db.session.add(event)
db.session.commit()

Updating related object with new event

The new event created has new “id” and “identifier”. This new “id” is added into foreign keys columns of the related object thus providing a relationship with the new event created.

for ticket in tickets:
   ticket_id = ticket.id
   db.session.expunge(ticket)  # expunge the object from session
   make_transient(ticket)
   ticket.event_id = event.id
   delattr(ticket, 'id')
   db.session.add(ticket)
   db.session.commit()

Finishing up

The last step of Updating related objects is repeated for all related objects to create the copy. Thus a new event is created with all related objects copied with the single endpoint.

References

How to clone a sqlalchemy object
https://stackoverflow.com/questions/28871406/how-to-clone-a-sqlalchemy-db-object-with-new-primary-key