Category Archives: Programming & Development

SyntaxError: (unicode error) ‘unicodeescape’ codec can’t decode bytes in position truncated \UXXXXXXXX escape

This is a common error in python when you are reading from a file and have specified a file path String. You can usually fix this by placing an ‘r’ in the front of your string to change it to a raw string.

Before fix (code has the issue)

book = epub.read_epub("C:\Users\ronni\Desktop\ebook\Frankenstein.epub")

After fix (issue is resolved)

book = epub.read_epub(r"C:\Users\ronni\Desktop\ebook\Frankenstein.epub")

Explanation

Python interprets “\” within a string as a special escape character. In this example, the \U in position 2-3 is a unicode escape character. Adding ‘r’ instructs the python interpreter to instead treat the string as a raw literal.

References

https://blog.softhints.com/python/

https://www.askpython.com/python/string/python-raw-strings

https://docs.python.org/3/reference/lexical_analysis.html

Advertisement

heroku cheat sheet quick start

#note requires java 8. add the ppa repo for oracle and install this if you dont have it already. check your local with: java -version

#create auth token for subsequent commands
heroku login

#from within a local git cloned app root, to create a new dyno
heroku create

#deploy
git push heroku master

#check if dyno is running
heroku ps:scale web=1

#shortcut to open the dyno. if you are running these in a remote SSH you will need to X11 forward for this to work
#or simply, just open the address indicated after you push
#this uri is also visible using the logs command below
heroku open

#tail logs for running dyno. refresh the browser while you are viewing the app to verify new log entries added
heroku logs --tail

#procfile declares what command is executed at start of app. this is likely web.
#for windows: this is different. see heroku docs and make Procfile.windows and use heroku -f Procfile.windows
https://devcenter.heroku.com/articles/procfile

#general note regarding the “free-ness” of heroku on your first app
“By default, your app is deployed on a free dyno. Free dynos will sleep after a half hour of inactivity and they can be active (receiving traffic) for no more than 18 hours a day before going to sleep. If a free dyno is sleeping, and it hasn’t exceeded the 18 hours, any web request will wake it. This causes a delay of a few seconds for the first request upon waking. Subsequent requests will perform normally”

#pom.xml specifies dependencies

#system.properties in the case of java determines version of java to use

#pull in dependencies to target/dependency and store this in pom.xml
mvn clean install

#to start heroku locally
heroku local web

#install add-ons
heroku addons:create papertrail

#list add-ons
heroku addons

#open add-on logs
heroku addons:open papertrail

#test a command on a one-off dyno, e.g. run bash in local dyno
heroku run bash

#set environment variables
heroku config:set ENERGY="20 GeV"

#view environment variables
heroku config

#show database info, e.g. postgres
heroku pg

#run tsql commands
heroku pg:psql

References:
https://devcenter.heroku.com/articles/getting-started-with-java

javac invalid target release maven ant intellij netbeans eclipse

You may see the following error when running maven or ant command or from intellij:
javac: invalid target release: 1.8

This occurs because you are running a build target for a version of java that you do not have set as your default in JAVA_HOME.

If you do not have java 8 installed, in ubuntu this is as simple as:
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

This is similar in other flavors of linux, simply add the official oracle repo and install. Installation in Windows requires browsing to the JDK/JRE pages on Oracle and downloading/installing. Avoid OpenJDK. This is outside the context of this blog post, but you can google “openjdk vs oracle jdk performance” to learn more.

If you see this error in an IDE such as intellij, netbeans or eclipse, this can be fixed by setting the java compile in your Settings to Java 8. E.g. if the error above said 1.9, change this to Java 9.

If you are upgrading from 7 to 8, the last oracle install step when installing the package should set JAVA_HOME:
(ubuntu) sudo apt-get install oracle-java8-set-default

If the above install command or corresponding install script on your system does not work, to fix you can also manually set JAVA_HOME, such as:
export JAVA_HOME=/usr/lib/jvm/java-8-oracle

echo the current $JAVA_HOME and the new oracle java home should be alongside it.

In my scenario I actually ran the oracle provided install script but it failed to set the java home correctly so I still had to do this manually.

References:
http://stackoverflow.com/questions/28044461/javac-invalid-target-release-1-8-on-mac-when-executing-maven-command

http://tecadmin.net/install-oracle-java-8-jdk-8-ubuntu-via-ppa/#

solarized color theme

Parent site:
http://ethanschoonover.com/solarized

Vim (standalone or with pathogen):
https://github.com/altercation/vim-colors-solarized

Visual Studio (also in main repo):
https://github.com/leddt/visualstudio-colors-solarized

IntelliJ (also in main repo):
https://github.com/jkaving/intellij-colors-solarized

Putty:
Install the .reg files to use with new sessions or edit previous sessions following the provided instructions.
https://github.com/brantb/solarized/tree/master/putty-colors-solarized

After you are connected to remote machine, you may need to run:

export TERM=xterm-256color

vimrc settings

Located in ~/.vimrc

If it doesn’t exist, create it!

set smartindent
set tabstop=2
set shiftwidth=2
set expandtab

"enables mouse in all modes. supports click and scrollwheel. use shift+ins to paste from clipboard
set mouse=a

"for use with pathogen
syntax enable
filetype plugin indent on

"for use with solarized. may require export TERM=xterm-256color
set background=dark
colorscheme solarized

To see more info on these settings, from within vim:

:help smartindent

To format a file with mixed tabs and spaces, after applying the above settings, from within vim:

:%retab

Query Sql Version Microsoft MSSQL

    set nocount on
    go
    select @@version
    go
    select 'Edition: ' + convert(char(30), serverproperty('Edition'))
    go  
    select 'Product Version: ' + convert(char(20), serverproperty('ProductVersion'))
    go
    select 'Product Level: ' + convert(char(20),serverproperty('ProductLevel'))
    go
    set nocount off
    go

References
https://support.microsoft.com/kb/321185

http://blogs.msdn.com/b/euanga/archive/2008/01/23/confused-by-sql-server-version-numbers.aspx

ASP .NET Code Contracts Examples

//contract.requires
public void Withdraw(Account account, decimal amount)
{
	Contract.Requires(account != null, "account cannot be null");
	Contract.Requires(amount > 0, "amount must be greater than zero");
	...
}

//contract.ensures
public double CalculateDiscount(int originalAmount, string discountCode)
{
	Contract.Ensures(Contract.Result<double>() >= 0);

	return -32;
}

Reference
http://visualstudiomagazine.com/articles/2010/06/23/code-contracts.aspx

http://msdn.microsoft.com/en-us/library/dd264808(v=vs.110).aspx

TPL Task Parrallel Library .net 4.0 parallel programming multi thread threading task queue user work item

//simple parallel function call
Parallel.Invoke(() => somefunc1(), () => somefunc2());

//parallel foreach with two lists
new List<List> tasks = new List<List>()
{
  new List() { new DataSource1ProcessorA() },
  new List() { new DataSource2ProcessorA(), new DataSource2ProcessorB() }
}

Parallel.ForEach(tasks, (items) =>
{
    foreach (var item in items)
    {
        item.GetData();
        item.Process();
    }
});

//new task static factory method approach
Task.Factory.StartNew(() => { data.GetData(); data.Process(); })
.ContinueWith(t => Logger.Error("An exception occurred while processing. Check the inner exception for details", t.Exception),
TaskContinuationOptions.OnlyOnFaulted);

//explicit object oriented
   // Create a task and supply a user delegate by using a lambda expression. 
        Task taskA = new Task( () => Console.WriteLine("Hello from taskA."));
        // Start the task.
        taskA.Start();
        // Output a message from the calling thread.
        Console.WriteLine("Hello from thread '{0}'.", 
                          Thread.CurrentThread.Name);
        taskA.Wait();

//using task run
  Thread.CurrentThread.Name = "Main";
      // Define and run the task.
      Task taskA = Task.Run( () => Console.WriteLine("Hello from taskA."));
      // Output a message from the calling thread.
      Console.WriteLine("Hello from thread '{0}'.", 
                          Thread.CurrentThread.Name);
      taskA.Wait();

References
http://msdn.microsoft.com/en-us/library/dd460705.aspx

(added 20121023 good comprehensive resource with illustrative examples)
http://www.codeproject.com/Articles/362996/Multi-core-programming-using-Task-Parallel-Library

http://www.codeproject.com/KB/threads/ParallelTasks.aspx

Optimize Managed Code For Multi-Core Machines
http://msdn.microsoft.com/en-us/magazine/cc163340.aspx

good blog
http://www.codethinked.com/net-40-and-systemthreadingtasks

queueuserworkitem
http://msdn.microsoft.com/en-us/library/4yd16hza.aspx

threadpool class
http://msdn.microsoft.com/en-us/library/y5htx827.aspx

task class
http://msdn.microsoft.com/en-us/library/system.threading.tasks.task.aspx

task parrallelism
http://msdn.microsoft.com/en-us/library/dd537609.aspx

How to: Use Parallel.Invoke to Execute Parallel Operations
http://msdn.microsoft.com/en-us/library/dd460705.aspx

Custom DAL Class SQL ORM ASP .NET

(common.DataObject may be of your choosing or may simply replace with dynamic)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Data;
using System.Web.UI.WebControls;
using System.Web.UI;
using System.Data.SqlClient;

using AIS.Common; //this is a common namespace I use in examples throughout my site
using System.Reflection;
using System.Dynamic;

//TODO: consider returning ienumerable in sp return values for lazy eval vs .tolist immediate eval
namespace AIS.DAL.AppName
{
    public static class StoredProcedures
    {
        public delegate void ErrorHandler(Exception ex);
        /// <summary>
        /// If no custom error handling is bound to this event, exceptions will be thrown back up to the calling function.
        /// If custom handling is bound to this event, ensure it does not perform a redirect or kill the thread unless you intend to abort the procedural
        /// steps following the method/function call which threw the error.
        /// </summary>
        public static event ErrorHandler HandleError;

        #region Unique Procedures
        public static List<Common.DataObject> LoadUserSessions_All(dynamic o)
        {
            return ExecuteRead("an_get_db_fn1", o);
        }

        public static List<Common.DataObject> LoadUserSessionsDetails_LiveStream(dynamic o)
        {
            return ExecuteRead("an_get_db_fn2", o);
        }

        public static List<Common.DataObject> LoadUserSessionsDetails_Live(dynamic o)
        {
            return ExecuteRead("an_get_db_fn3", o);
        }

        public static int LogChat()
        {
            return ExecuteScalar("an_get_db_fn4", null);
        }

        public static int LogError()
        {
            return ExecuteScalar("an_get_db_fn5", null);
        }
        #endregion

        //TODO: consider hiding from external assemblies which would require strong mappings above
        #region Execution Logic
        public static List<Common.DataObject> ExecuteRead(string procedurename, dynamic param)
        {
            try
            {
                SqlDataSource sds = new SqlDataSource();
                sds.ConnectionString = ConfigValues.TrainingPortalConnectionString;
                sds.SelectCommandType = SqlDataSourceCommandType.StoredProcedure;
                sds.SelectCommand = procedurename;

                if (param != null)
                {
                    foreach (PropertyInfo pi in param.GetType().GetProperties())
                    {
                        object pval = pi.GetValue(param, null);
                        if (pval != null)
                        {
                            sds.SelectParameters.Add(pi.Name, pval.ToString());
                        }
                    }
                }

                List<Common.DataObject> results = new List<Common.DataObject>();
                //sds.Select(new DataSourceSelectArguments()).Cast<DataRowView>().ToList().ForEach(o => Load_AddResult<dynamic>(o, ref results));
                sds.Select(new DataSourceSelectArguments()).Cast<DataRowView>().ToList().ForEach(o => Load_AddResult<Common.DataObject>(o, ref results));

                return results;
            }
            catch (Exception ex)
            {
                HandleError_Condensed(ex);
                return null;
            }
        }

        public static void Load_AddResult<t>(Common.DataObject o, ref List<t> results)
        {
            try
            {
                t r = (t)Activator.CreateInstance(typeof(t));

                foreach (PropertyInfo pi in typeof(t).GetProperties())
                {
                    object v = o[pi.Name].ToString();
                    Type pt = Type.GetType(pi.PropertyType.FullName);
                    //try { pi.SetValue(r, Convert.ChangeType(v, pt), null); }
                    //catch (Exception ex) { HandleError_Condensed(ex); }

                    o.Add(pi.Name, Convert.ChangeType(v, pt));
                }

                results.Add(r);
            }
            catch (Exception ex)
            {
                HandleError_Condensed(ex);
            }
        }

        //public static void Load_AddResult<t>(dynamic o, ref List<t> results)
        //{
        //    try
        //    {
        //        t r = (t)Activator.CreateInstance(typeof(t));

        //        foreach (PropertyInfo pi in typeof(t).GetProperties())
        //        {
        //            object v = o[pi.Name].ToString();
        //            Type pt = Type.GetType(pi.PropertyType.FullName);
        //            try { pi.SetValue(r, Convert.ChangeType(v, pt), null); }
        //            catch (Exception ex) { HandleError_Condensed(ex); }
        //        }

        //        results.Add(r);
        //    }
        //    catch (Exception ex)
        //    {
        //        HandleError_Condensed(ex);
        //    }
        //}

        public static void ExecuteNonScalar(string procedurename, dynamic param)
        {
            try
            {
                ExecuteScalar(procedurename, param);
            }
            catch (Exception ex)
            {
                HandleError_Condensed(ex);
            }
        }

        public static int ExecuteScalar(string procedurename, dynamic param)
        {
            try
            {
                SqlDataSource sds = new SqlDataSource();
                sds.ConnectionString = ConfigValues.TrainingPortalConnectionString;
                sds.UpdateCommandType = SqlDataSourceCommandType.StoredProcedure;
                sds.UpdateCommand = procedurename;

                if (param != null)
                {
                    foreach (PropertyInfo pi in param.GetType().GetProperties())
                    {
                        object pval = pi.GetValue(param, null);
                        if (pval != null)
                        {
                            sds.SelectParameters.Add(pi.Name, pval.ToString());
                        }
                    }
                }

                return sds.Update();
            }
            catch (Exception ex)
            {
                HandleError_Condensed(ex);
                return 1; //1 signifies error in tsql
            }
        }
        #endregion

        private static void HandleError_Condensed(Exception ex)
        {
            if (HandleError != null) { HandleError(ex); } else { throw new Exception(ex.Message, ex); } 
        }
    }
}

Custom AWS S3 Helper Class AWSSDK Wrapper

Implements some very commonly used AWS S3 functionality. (need to merge with my other AWS wrapper classes, Route53, EC2, etc)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;

using System.IO;
using System.Collections.Specialized;
using System.Configuration;

//uses AWSSDK.dll from amazon
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;

using System.Xml.Linq;
using System.Xml;

using System.Data;

namespace AIS.Common
{
    public static class AWSHelper
    {
        private static List<S3Bucket> LoadS3Buckets()
        {
            System.Collections.Specialized.NameValueCollection appConfig = System.Configuration.ConfigurationManager.AppSettings;
            using (var s3client = Amazon.AWSClientFactory.CreateAmazonS3Client(ConfigValues.AWSAccessKey, ConfigValues.AWSSecretKey))
            {
                return s3client.ListBuckets().Buckets;
            }
        }

        private static List<S3Object> LoadS3Objects(string bucketname)
        {
            System.Collections.Specialized.NameValueCollection appConfig = System.Configuration.ConfigurationManager.AppSettings;
            using (var s3client = Amazon.AWSClientFactory.CreateAmazonS3Client(ConfigValues.AWSAccessKey, ConfigValues.AWSSecretKey))
            {
                return s3client.ListObjects(new ListObjectsRequest() { BucketName = bucketname }).S3Objects;
            }
        }

        private static void LoadS3File(string bucketname, string keyname, HttpResponse response, string contenttype)
        {
            NameValueCollection appConfig = ConfigurationManager.AppSettings;


            using (var s3client = Amazon.AWSClientFactory.CreateAmazonS3Client(ConfigValues.AWSAccessKey, ConfigValues.AWSSecretKey))
            {

                GetObjectRequest s3request = new GetObjectRequest()
                    .WithBucketName(bucketname).WithKey(keyname);

                using (GetObjectResponse s3response = s3client.GetObject(s3request))
                {
                    string title = s3response.Metadata["x-amz-meta-title"];

                    response.Clear();

                    //Response.Write(string.Format("The object's title is {0}", title));
                    //Response.AddHeader
                    //Response.ContentType="application/swf";
                    ////Response.ContentType="contenttype";

                    response.ContentType = s3response.ContentType; //s3response.Headers["Content-Length"];
                    long filesize = s3response.ContentLength;

                    byte[] buffer = new byte[(int)filesize];

                    response.BinaryWrite(ConvertStreamToBytes(s3response.ResponseStream, filesize));

                    response.Flush();
                    response.Close();
                }
            }
        }

        public static string GetS3UrlToVideo(string bucketname, string keyname)
        {
            System.Collections.Specialized.NameValueCollection appConfig = System.Configuration.ConfigurationManager.AppSettings;
            string url = "";
            using (var s3client = Amazon.AWSClientFactory.CreateAmazonS3Client(ConfigValues.AWSAccessKey, ConfigValues.AWSSecretKey))
            {
                Amazon.S3.Model.GetPreSignedUrlRequest request = new Amazon.S3.Model.GetPreSignedUrlRequest()
                    .WithBucketName(bucketname)
                    .WithKey(keyname)
                    .WithProtocol(Amazon.S3.Model.Protocol.HTTP)
                    .WithVerb(HttpVerb.GET)
                .WithExpires(DateTime.Now.AddMinutes(ConfigValues.VideoURLExpiration));

                Amazon.S3.Model.GetPreSignedUrlRequest r = new GetPreSignedUrlRequest();

                url = s3client.GetPreSignedURL(request);

                url= "https://s3.amazonaws.com/" + bucketname + keyname;
            }

            //return System.Xml.XmlConvert.EncodeName(url);
            return url;
        }

        public static byte[] ConvertStreamToBytes(Stream input, long filesize)
        {
            byte[] buffer = new byte[(int)filesize];
            using (MemoryStream ms = new MemoryStream())
            {
                int read;
                while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
                {
                    ms.Write(buffer, 0, read);
                }
                return ms.ToArray();
            }
        }
    }
}