A simple mistaken occurred leveraging spark in python multiprocessing

Look at this snippet first: It looks fine at the first glance. However, after the validation, the output was incomplete in delta table. At the end, the issue happens in the df_test, which is not a local variable in a function. So, when it ran as multicore, df_test was overwritten. The best way to avoid… Continue reading A simple mistaken occurred leveraging spark in python multiprocessing

Learn Django with me(part 3)

Handle view and templates View consist of a set of functions which handle the different url request with the specific url pettarns. And it returns either of HttpResponse or Http404. Firstly, let’s update webapp/views.py: from django.shortcuts import render from .models import Question def index(request): # get the lastest 5 questions latest_question_list = Question.objects.order_by(‘-pub_date’)[:5] # create… Continue reading Learn Django with me(part 3)

Learn Django with me(part 2)

Change Database Setting Open up mysite/settings.py, find snippet as blew: DATABASES = { ‘default’: { ‘ENGINE’: ‘django.db.backends.sqlite3’, ‘NAME’: os.path.join(BASE_DIR, ‘db.sqlite3′), } } Here you can change your database to others if needed * ENGINE – Either ‘django.db.backends.sqlite3’, ‘django.db.backends.postgresql’, ‘django.db.backends.mysql’, or ‘django.db.backends.oracle’. * NAME – Name of database We can aslo change the time zone in… Continue reading Learn Django with me(part 2)